From owner-freebsd-stable@FreeBSD.ORG Thu Jan 27 20:03:03 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46284106566C for ; Thu, 27 Jan 2011 20:03:03 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id EDF918FC19 for ; Thu, 27 Jan 2011 20:03:02 +0000 (UTC) Received: by gwj21 with SMTP id 21so824713gwj.13 for ; Thu, 27 Jan 2011 12:03:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=pGeSt05Fm5mPVvgFYr5VNoHugJlyM+K8wOkLiefnLCs=; b=HH5wfaqQYg8qNgTignRT/wE5LAACJ+O4OZ4hjec+J1kZaqyXp7XELyoo+ENCM5VSXR WZWCvIWVs//dTlNMeEowVK6CyB69yYiBes0DQsoXGmSBsT7SOQOlPrykAOW9wNjFaoYB lFLWMcHTghEhxKHWIX85iOhqKxP2WN3dkBdTg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=xAEfzd//q+dPNZLx3hqnxWEivdJCV1jtbRfG+f7cXxOFZN66srwbFnDO9qxPoE5nXx sGvGETFUEeQejyDOCmtQc2wAPh7FWVJz/U2l1y7RudDlTa10KKSCn+mS2AF4QUJw9Wx8 PozhWhRbw6VUawzRp77O44SChLnvV1aArsYNQ= MIME-Version: 1.0 Received: by 10.236.108.177 with SMTP id q37mr3091801yhg.11.1296158538636; Thu, 27 Jan 2011 12:02:18 -0800 (PST) Received: by 10.147.171.17 with HTTP; Thu, 27 Jan 2011 12:02:18 -0800 (PST) In-Reply-To: <20110127195741.GA40449@icarus.home.lan> References: <4D41417A.20904@my.gd> <1DB50624F8348F48840F2E2CF6040A9D014BEB8833@orsmsx508.amr.corp.intel.com> <4D41B197.6070308@my.gd> <201101280146.57028.wmn@siberianet.ru> <4D41C9FC.10503@my.gd> <20110127195741.GA40449@icarus.home.lan> Date: Thu, 27 Jan 2011 12:02:18 -0800 Message-ID: From: Jack Vogel To: Jeremy Chadwick Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Sergey Lobanov , "freebsd-stable@freebsd.org" , "freebsd-pf@freebsd.org" Subject: Re: High interrupt rate on a PF box + performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 20:03:03 -0000 If you go to 8.2 and the latest driver you will get better stats also, ahem... Jack On Thu, Jan 27, 2011 at 11:57 AM, Jeremy Chadwick wrote: > On Thu, Jan 27, 2011 at 08:39:40PM +0100, Damien Fleuriot wrote: > > > > > > On 1/27/11 7:46 PM, Sergey Lobanov wrote: > > > =F7 =D3=CF=CF=C2=DD=C5=CE=C9=C9 =CF=D4 =F0=D1=D4=CE=C9=C3=C1 28 =D1= =CE=D7=C1=D2=D1 2011 00:55:35 =C1=D7=D4=CF=D2 Damien Fleuriot > =CE=C1=D0=C9=D3=C1=CC: > > >> On 1/27/11 6:41 PM, Vogel, Jack wrote: > > >>> Jeremy is right, if you have a problem the first step is to try the > > >>> latest code. > > >>> > > >>> However, when I look at the interrupts below I don't see what the > problem > > >>> is? The Broadcom seems to have about the same rate, it just doesn't > have > > >>> MSIX (multiple vectors). > > >>> > > >>> Jack > > >> > > >> My main concern is that the CPU %interrupt is quite high, also, we > seem > > >> to be experiencing input errors on the interfaces. > > > Would you show igb tuning which is done in loader.conf and output of > sysctl > > > dev.igb.0? > > > Did you rise number of igb descriptors such as: > > > hw.igb.rxd=3D4096 > > > hw.igb.txd=3D4096 ? > > > > There is no tuning at all on our part in the loader's conf. > > > > Find below the sysctls: > > > > # sysctl -a |grep igb > > dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 > > dev.igb.0.%driver: igb > > dev.igb.0.%location: slot=3D0 function=3D0 > > dev.igb.0.%pnpinfo: vendor=3D0x8086 device=3D0x10d6 subvendor=3D0x8086 > > subdevice=3D0x145a class=3D0x020000 > > dev.igb.0.%parent: pci14 > > dev.igb.0.debug: -1 > > dev.igb.0.stats: -1 > > dev.igb.0.flow_control: 3 > > dev.igb.0.enable_aim: 1 > > dev.igb.0.low_latency: 128 > > dev.igb.0.ave_latency: 450 > > dev.igb.0.bulk_latency: 1200 > > dev.igb.0.rx_processing_limit: 100 > > dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 > > dev.igb.1.%driver: igb > > dev.igb.1.%location: slot=3D0 function=3D1 > > dev.igb.1.%pnpinfo: vendor=3D0x8086 device=3D0x10d6 subvendor=3D0x8086 > > subdevice=3D0x145a class=3D0x020000 > > dev.igb.1.%parent: pci14 > > dev.igb.1.debug: -1 > > dev.igb.1.stats: -1 > > dev.igb.1.flow_control: 3 > > dev.igb.1.enable_aim: 1 > > dev.igb.1.low_latency: 128 > > dev.igb.1.ave_latency: 450 > > dev.igb.1.bulk_latency: 1200 > > dev.igb.1.rx_processing_limit: 100 > > I'm not aware of how to tune igb(4), so the advice Sergey gave you may > be applicable. You'll need to schedule downtime to adjust those > tunables however (since a reboot will be requried). > > I also reviewed the munin graphs. I don't see anything necessarily > wrong. However, you omitted yearly graphs for the network interfaces. > Why I care about that: > > The pf state table (yearly) graph basically correlates with the CPU > usage (yearly) graph, and I expect that the yearly network graphs would > show a similar trend: an increase in your overall traffic over the > course of a year. > > What I'm trying to figure out is what you're concerned about. You are > in fact pushing anywhere between 60-120MBytes/sec across these > interfaces. Given those numbers, I'm not surprised by the ""high"" > interrupt usage. > > Graphs of this nature usually indicate that you're hitting a > "bottleneck" (for lack of better word) where you're simply doing "too > much" with a single machine (given its network throughput). The machine > is spending a tremendous amount of CPU time handling network traffic, > and equally as much with regards to the pf usage. > > If you want my opinion based on the information I have so far, it's > this: you need to scale your infrastructure. You can no longer rely on > a single machine to handle this amount of traffic. > > As for the network errors you see -- to get low-level NIC and driver > statistics, you'll need to run "sysctl dev.igb.X.stats=3D1" then run > "dmesg" and look at the numbers shown (the sysctl command won't output > anything itself). This may help indicate where the packets are being > lost. You should also check the interface counters on the switch which > these interfaces are connected to. I sure hope it's a managed switch > which can give you those statistics. > > Hope this helps, or at least acts as food for thought. > > -- > | Jeremy Chadwick jdc@parodius.com | > | Parodius Networking http://www.parodius.com/ | > | UNIX Systems Administrator Mountain View, CA, USA | > | Making life hard for others since 1977. PGP 4BD6C0CB | > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >