From owner-freebsd-stable@FreeBSD.ORG Thu Jan 27 09:57:26 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AAAEE1065673; Thu, 27 Jan 2011 09:57:26 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 02DD88FC0C; Thu, 27 Jan 2011 09:57:24 +0000 (UTC) Received: by bwz12 with SMTP id 12so2249634bwz.13 for ; Thu, 27 Jan 2011 01:57:23 -0800 (PST) Received: by 10.204.56.3 with SMTP id w3mr1368730bkg.60.1296122242330; Thu, 27 Jan 2011 01:57:22 -0800 (PST) Received: from dfleuriot.technique-admin.paris.hi-media-techno.com ([83.167.62.196]) by mx.google.com with ESMTPS id 12sm7986486bki.19.2011.01.27.01.57.16 (version=SSLv3 cipher=RC4-MD5); Thu, 27 Jan 2011 01:57:18 -0800 (PST) Message-ID: <4D41417A.20904@my.gd> Date: Thu, 27 Jan 2011 10:57:14 +0100 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-pf@freebsd.org Subject: High interrupt rate on a PF box + performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Jan 2011 09:57:26 -0000 Hello list, I have a problem with interrupts, network cards, and PF performance. We have 2 firewalls running FreeBSD 8.0 for the current master and FreeBSD 8.1 for the backup host, which I upgraded just yesterday. The servers use CARP for redundancy. These are rather busy boxes which run PF and nginx as a reverse proxy. As you will see below, we're getting a "high" %interrupt CPU usage, which seems to come mostly from the NICs. I'm wondering if there is any way to optimize the box's performance and reduce the interrupts rate or the CPU usage ? Also, we've noticed a sharp drop in CPU usage since we've disabled pfsync, but we'd rather keep it now wouldn't we ? Last, we seem to get input errors on the NICs, although the switch ports report not a single layer 2 error in over a year. I'm wondering what counts as a NIC input error ? Hardware is as follows: CPU -- CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2496.25-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Features=0xbfebfbff Features2=0xce3bd AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 MEM -- real memory = 2147483648 (2048 MB) avail memory = 2057293824 (1961 MB) NICs -- bce0: mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7 bce1: mem 0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3 igb0: port 0xdce0-0xdcff mem 0xfd0e0000-0xfd0fffff,0xfce00000-0xfcffffff,0xfd0dc000-0xfd0dffff irq 18 at device 0.0 on pci14 igb0: Using MSIX interrupts with 3 vectors Find below different outputs from the current master running FreeBSD 8.0-RELEASE-p2 systat -v --- 3 users Load 0.41 0.31 0.29 Jan 26 18:59 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 143036 8152 836392 11188 1262556 count All 168224 10420 1074653k 31172 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 36163 total 47 105k 76 2077 28k 223 zfod ata0 irq14 ozfod mfi0 irq16 4.3%Sys 28.1%Intr 3.0%User 0.0%Nice 64.7%Idle %ozfod uhci0 uhci | | | | | | | | | | | daefr 1998 cpu0: time ==++++++++++++++>> prcfr 9428 bce0 256 33 dtbuf totfr 12931 igb0 257 Namei Name-cache Dir-cache 100000 desvn react 5791 igb0 258 Calls hits % hits % 70448 numvn pdwak igb0 259 24988 frevn pdpgs igb1 260 intrn 1 igb1 261 Disks mfid0 372392 wire igb1 262 KB/t 0.00 62336 act 20 bce1 269 tps 0 323720 inact 1998 cpu1: time MB/s 0.00 292 cache 1998 cpu2: time %busy 0 1262264 free 1998 cpu3: time 218272 buf vmstat -i --- interrupt total rate irq14: ata0 36 0 irq16: mfi0 353244 1 irq21: uhci0 uhci+ 461504 1 cpu0: timer 615183815 1996 irq256: bce0 1015412475 3295 irq257: igb0 1067318584 3464 irq258: igb0 695648752 2258 irq259: igb0 2 0 irq260: igb1 11503857 37 irq261: igb1 506598 1 irq262: igb1 69 0 irq269: bce1 790820 2 cpu1: timer 615183757 1996 cpu2: timer 615197165 1996 cpu3: timer 615197165 1996 Total 5252757843 17050 pf status (159 filter rules, 17 nat/rdr rules) --- # pfctl -si Status: Enabled for 3 days 13:34:56 Debug: Urgent Interface Stats for igb0 IPv4 IPv6 Bytes In 487209136643 384 Bytes Out 687158173727 0 Packets In Passed 1967249106 0 Blocked 6183860 6 Packets Out Passed 2018192359 0 Blocked 686901 0 State Table Total Rate current entries 25428 searches 9006187476 29231.8/s inserts 679746853 2206.3/s removals 679721425 2206.2/s Counters match 686988143 2229.8/s bad-offset 0 0.0/s fragment 56 0.0/s short 0 0.0/s normalize 171 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 1 0.0/s proto-cksum 13916 0.0/s state-mismatch 220169 0.7/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 1812 0.0/s synproxy 0 0.0/s Regards, -- dfl