Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 27 Jan 2011 10:57:14 +0100
From:      Damien Fleuriot <ml@my.gd>
To:        "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org>
Cc:        freebsd-pf@freebsd.org
Subject:   High interrupt rate on a PF box + performance
Message-ID:  <4D41417A.20904@my.gd>

next in thread | raw e-mail | index | archive | help
Hello list,



I have a problem with interrupts, network cards, and PF performance.



We have 2 firewalls running FreeBSD 8.0 for the current master and
FreeBSD 8.1 for the backup host, which I upgraded just yesterday.

The servers use CARP for redundancy.

These are rather busy boxes which run PF and nginx as a reverse proxy.



As you will see below, we're getting a "high" %interrupt CPU usage,
which seems to come mostly from the NICs.

I'm wondering if there is any way to optimize the box's performance and
reduce the interrupts rate or the CPU usage ?

Also, we've noticed a sharp drop in CPU usage since we've disabled
pfsync, but we'd rather keep it now wouldn't we ?

Last, we seem to get input errors on the NICs, although the switch ports
report not a single layer 2 error in over a year.
I'm wondering what counts as a NIC input error ?



Hardware is as follows:


CPU
--
CPU: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz (2496.25-MHz
K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x10676  Stepping = 6

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant

ACPI APIC Table: <DELL   PE_SC3  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3


MEM
--
real memory  = 2147483648 (2048 MB)
avail memory = 2057293824 (1961 MB)



NICs
--
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3
igb0: <Intel(R) PRO/1000 Network Connection version - 1.7.3> port
0xdce0-0xdcff mem
0xfd0e0000-0xfd0fffff,0xfce00000-0xfcffffff,0xfd0dc000-0xfd0dffff irq 18
at device 0.0 on pci14
igb0: Using MSIX interrupts with 3 vectors


Find below different outputs from the current master running FreeBSD
8.0-RELEASE-p2





systat -v
---
    3 users    Load  0.41  0.31  0.29                  Jan 26 18:59

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP
PAGER
        Tot   Share      Tot    Share    Free           in   out     in
  out
Act  143036    8152   836392    11188 1262556  count
All  168224   10420 1074653k    31172          pages
Proc:                                                            Interrupts
  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        cow   36163 total
             47      105k   76 2077  28k  223             zfod
ata0 irq14
                                                          ozfod
mfi0 irq16
 4.3%Sys  28.1%Intr  3.0%User  0.0%Nice 64.7%Idle        %ozfod
uhci0 uhci
|    |    |    |    |    |    |    |    |    |    |       daefr  1998
cpu0: time
==++++++++++++++>>                                        prcfr  9428
bce0 256
                                        33 dtbuf          totfr 12931
igb0 257
Namei     Name-cache   Dir-cache    100000 desvn          react  5791
igb0 258
   Calls    hits   %    hits   %     70448 numvn          pdwak
igb0 259
                                     24988 frevn          pdpgs
igb1 260
                                                          intrn     1
igb1 261
Disks mfid0                                        372392 wire
igb1 262
KB/t   0.00                                         62336 act      20
bce1 269
tps       0                                        323720 inact  1998
cpu1: time
MB/s   0.00                                           292 cache  1998
cpu2: time
%busy     0                                       1262264 free   1998
cpu3: time
                                                   218272 buf



vmstat -i
---
interrupt                          total       rate
irq14: ata0                           36          0
irq16: mfi0                       353244          1
irq21: uhci0 uhci+                461504          1
cpu0: timer                    615183815       1996
irq256: bce0                  1015412475       3295
irq257: igb0                  1067318584       3464
irq258: igb0                   695648752       2258
irq259: igb0                           2          0
irq260: igb1                    11503857         37
irq261: igb1                      506598          1
irq262: igb1                          69          0
irq269: bce1                      790820          2
cpu1: timer                    615183757       1996
cpu2: timer                    615197165       1996
cpu3: timer                    615197165       1996
Total                         5252757843      17050




pf status (159 filter rules, 17 nat/rdr rules)
---
# pfctl -si
Status: Enabled for 3 days 13:34:56           Debug: Urgent

Interface Stats for igb0              IPv4             IPv6
  Bytes In                    487209136643              384
  Bytes Out                   687158173727                0
  Packets In
    Passed                      1967249106                0
    Blocked                        6183860                6
  Packets Out
    Passed                      2018192359                0
    Blocked                         686901                0

State Table                          Total             Rate
  current entries                    25428
  searches                      9006187476        29231.8/s
  inserts                        679746853         2206.3/s
  removals                       679721425         2206.2/s
Counters
  match                          686988143         2229.8/s
  bad-offset                             0            0.0/s
  fragment                              56            0.0/s
  short                                  0            0.0/s
  normalize                            171            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              1            0.0/s
  proto-cksum                        13916            0.0/s
  state-mismatch                    220169            0.7/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                           1812            0.0/s
  synproxy                               0            0.0/s



Regards,

--
dfl



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D41417A.20904>