From owner-freebsd-stable@FreeBSD.ORG Thu Jan 27 17:37:50 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6873C1065679 for ; Thu, 27 Jan 2011 17:37:50 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta07.westchester.pa.mail.comcast.net (qmta07.westchester.pa.mail.comcast.net [76.96.62.64]) by mx1.freebsd.org (Postfix) with ESMTP id 142918FC08 for ; Thu, 27 Jan 2011 17:37:49 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by qmta07.westchester.pa.mail.comcast.net with comcast id 0hbD1g0080bG4ec57hdqlh; Thu, 27 Jan 2011 17:37:50 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta03.westchester.pa.mail.comcast.net with comcast id 0hdS1g01s2tehsa3PhdVHT; Thu, 27 Jan 2011 17:37:45 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 28F6C9B422; Thu, 27 Jan 2011 09:37:23 -0800 (PST) Date: Thu, 27 Jan 2011 09:37:23 -0800 From: Jeremy Chadwick To: Damien Fleuriot Message-ID: <20110127173723.GA36846@icarus.home.lan> References: <4D41417A.20904@my.gd> <20110127172724.GA36587@icarus.home.lan> <4D41ABF1.1010405@my.gd> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D41ABF1.1010405@my.gd> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, "Vogel, Jack" , freebsd-pf@freebsd.org Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 17:37:50 -0000 On Thu, Jan 27, 2011 at 06:31:29PM +0100, Damien Fleuriot wrote: > > > On 1/27/11 6:27 PM, Jeremy Chadwick wrote: > > On Thu, Jan 27, 2011 at 10:57:14AM +0100, Damien Fleuriot wrote: > >> Hello list, > >> > >> I have a problem with interrupts, network cards, and PF performance. > >> > >> We have 2 firewalls running FreeBSD 8.0 for the current master and > >> FreeBSD 8.1 for the backup host, which I upgraded just yesterday. > >> > >> [...] > >> > >> vmstat -i > >> --- > >> interrupt total rate > >> irq14: ata0 36 0 > >> irq16: mfi0 353244 1 > >> irq21: uhci0 uhci+ 461504 1 > >> cpu0: timer 615183815 1996 > >> irq256: bce0 1015412475 3295 > >> irq257: igb0 1067318584 3464 > >> irq258: igb0 695648752 2258 > >> irq259: igb0 2 0 > >> irq260: igb1 11503857 37 > >> irq261: igb1 506598 1 > >> irq262: igb1 69 0 > >> irq269: bce1 790820 2 > >> cpu1: timer 615183757 1996 > >> cpu2: timer 615197165 1996 > >> cpu3: timer 615197165 1996 > >> Total 5252757843 17050 > > > > There are changes to the igb(4) driver which are in RELENG_8 (8-STABLE), > > and some which will be in the upcoming 8.2-RELEASE, which may address > > this. Jack Vogel of Intel would be able to confirm for sure; CC'ing him > > here. > > > > Could you please provide output from the following commands? > > > > * pciconf -lvcb (only include igbX entries, thanks) > > * sysctl -a | grep msi > > > > Thanks. > > > > I can't help with the CARP-related issues or other stuff you're > > experiencing. These issues may all be separate problems, hard to say. > > > > > igb0@pci0:14:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 > rev=0x02 hdr=0x00 > igb1@pci0:14:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 > rev=0x02 hdr=0x00 > igb2@pci0:15:0:0: class=0x020000 card=0x145a8086 chip=0x10d68086 > rev=0x02 hdr=0x00 > igb3@pci0:15:0:1: class=0x020000 card=0x145a8086 chip=0x10d68086 > rev=0x02 hdr=0x00 What you did here was "pciconf -lvcb | grep igb", or something equivalent. There is output after each of these entries which is highly relevant. Example for an emX device: em1@pci0:15:0:0: class=0x020000 card=0x109a15d9 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = 'Intel PRO/1000 PL Network Adaptor (82573L)' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xdc300000, size 131072, enabled bar [18] = type I/O Port, range 32, base 0x3000, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) This is the sort of output we're looking for. > hw.bce.msi_enable: 1 > hw.pci.honor_msi_blacklist: 1 > hw.pci.enable_msix: 1 > hw.pci.enable_msi: 1 > > > igb0: flags=8943 metric > 0 mtu 1500 > options=13b > ether 00:1b:21:12:ec:38 > inet [snip] netmask 0xffffffc0 broadcast [snip] > media: Ethernet autoselect (1000baseT ) > status: active > > igb1: flags=8843 metric 0 mtu 1500 > options=13b > ether 00:1b:21:12:ec:39 > inet 10.0.0.252 netmask 0xffffff00 broadcast 10.0.0.255 > media: Ethernet autoselect (1000baseT ) > status: active > > > Here you go :) > > Note that the igb2 and 3 interfaces are unused, unplugged, unconfigured -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |