From owner-freebsd-net@FreeBSD.ORG Fri Apr 26 11:30:45 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 699C413A for ; Fri, 26 Apr 2013 11:30:45 +0000 (UTC) (envelope-from nodens2099@gmail.com) Received: from mail-ee0-f44.google.com (mail-ee0-f44.google.com [74.125.83.44]) by mx1.freebsd.org (Postfix) with ESMTP id 019E1124B for ; Fri, 26 Apr 2013 11:30:44 +0000 (UTC) Received: by mail-ee0-f44.google.com with SMTP id t10so1715074eei.31 for ; Fri, 26 Apr 2013 04:30:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=wtlZWnb38LqOwMIe7txZoPpTql/ZA8/ULpUKWLOKIDY=; b=c96cYkubqxvwcxx3bkzRem8TS37KPQh24JRo4F2z0L7K5H9MTn1OmIFVtlZgDTcvnQ 7iqFWGMWU/AdRf8aw66Jh85EMATvRWdSnRGDJdEoCZ1k2QLIUCHpxh6lwyoP/fgQ9Kha 3j2dG4SlykYSVDrWejXoxavEFvj7JkUnCcJ5xzaaVvPdRquIOURmz1QfufyvnUJWfYNu GCjga/V32RQZAAyhz6YmzIQYdnZs57ZGUciRBbPYbaQ/gzydC7fAJ7R7++tyI6vkpqKL BDMkkd6jtqtxk2Ors9EFlfEMHT/je0+mCtn0XyDkZsj4NoxYT9KEi2tCuvIL9Zsstv3L yqWw== X-Received: by 10.14.115.131 with SMTP id e3mr12529280eeh.43.1366975837765; Fri, 26 Apr 2013 04:30:37 -0700 (PDT) Received: from 11A0116.eolas.loc ([178.237.98.13]) by mx.google.com with ESMTPSA id k43sm15512709een.2.2013.04.26.04.30.36 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 26 Apr 2013 04:30:36 -0700 (PDT) Message-ID: <517A657B.7060003@gmail.com> Date: Fri, 26 Apr 2013 13:31:07 +0200 From: =?UTF-8?B?IkNsw6ltZW50IEhlcm1hbm4gKG5vZGVucyki?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130329 Thunderbird/17.0.5 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: High CPU interrupt load on intel I350T4 with igb on 8.3 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Apr 2013 11:30:45 -0000 Hi list, We use pf+ALTQ for trafic shaping on some routers. We are switching to new servers : Dell PowerEdge R620 with 2 8-cores Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 (quad port) using igb driver. The old hardware is using em driver, the CPU load is high but mostly due to kernel and a large pf ruleset. On the new hardware, we see high CPU Interrupt load (up to 95%), even though there is not much trafic currently (peaks about 150Mbps and 40Kpps). All queues are used and binded to a cpu according to top, but a lot of CPU time is spent on igb queues (interrupt or wait). The load is fine when we stay below 20Kpps. We see no mbuf shortage, no dropped packet, but there is little margin left on CPU time (about 25% idle at best, most of CPU time is spent on interrupts), which is disturbing. We have done some tuning, but to no avail : sysctl.conf : # mbufs kern.ipc.nmbclusters=65536 # Sockets kern.ipc.somaxconn=8192 net.inet.tcp.delayed_ack=0 net.inet.tcp.sendspace=65535 net.inet.udp.recvspace=65535 net.inet.udp.maxdgram=57344 net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 # IGB dev.igb.0.rx_processing_limit=4096 dev.igb.1.rx_processing_limit=4096 dev.igb.2.rx_processing_limit=4096 dev.igb.3.rx_processing_limit=4096 /boot/loader.conf : vm.kmem_size=1G hw.igb.max_interrupt_rate="32000" # maximum number of interrupts/sec generated by single igb(4) (default 8000) hw.igb.txd="2048" # number of transmit descriptors allocated by the driver (2048 limit) hw.igb.rxd="2048" # number of receive descriptors allocated by the driver (2048 limit) hw.igb.rx_process_limit="1000" # maximum number of received packets to process at a time, The default of 100 is # too low for most firewalls. (-1 means unlimited) Kernel HZ is 1000. The IGB /boot/loader.conf tuning was our last attempt, it didn't change anything. Does anyone have any pointer ? How could we lower CPU interrupt load ? should we set hw.igb.max_interrupt_rate lower instead of higher ? From what we saw here and there, we should be able to do much better with this hardware. relevant sysctl (igb1 and igb2 only, other interfaces are unused) : sysctl dev.igb | grep -v ": 0$" dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.3.1 dev.igb.1.%driver: igb dev.igb.1.%location: slot=0 function=1 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x8086 subdevice=0x5001 class=0x020000 dev.igb.1.%parent: pci5 dev.igb.1.nvm: -1 dev.igb.1.enable_aim: 1 dev.igb.1.fc: 3 dev.igb.1.rx_processing_limit: 4096 dev.igb.1.eee_disabled: 1 dev.igb.1.link_irq: 2 dev.igb.1.device_control: 1209795137 dev.igb.1.rx_control: 67141658 dev.igb.1.interrupt_mask: 4 dev.igb.1.extended_int_mask: 2147483981 dev.igb.1.fc_high_water: 33168 dev.igb.1.fc_low_water: 33152 dev.igb.1.queue0.interrupt_rate: 71428 dev.igb.1.queue0.txd_head: 1318 dev.igb.1.queue0.txd_tail: 1318 dev.igb.1.queue0.tx_packets: 84663594 dev.igb.1.queue0.rxd_head: 717 dev.igb.1.queue0.rxd_tail: 715 dev.igb.1.queue0.rx_packets: 43899597 dev.igb.1.queue0.rx_bytes: 8905556030 dev.igb.1.queue1.interrupt_rate: 90909 dev.igb.1.queue1.txd_head: 693 dev.igb.1.queue1.txd_tail: 693 dev.igb.1.queue1.tx_packets: 57543349 dev.igb.1.queue1.rxd_head: 1033 dev.igb.1.queue1.rxd_tail: 1032 dev.igb.1.queue1.rx_packets: 54821897 dev.igb.1.queue1.rx_bytes: 9944955108 dev.igb.1.queue2.interrupt_rate: 100000 dev.igb.1.queue2.txd_head: 350 dev.igb.1.queue2.txd_tail: 350 dev.igb.1.queue2.tx_packets: 62320990 dev.igb.1.queue2.rxd_head: 1962 dev.igb.1.queue2.rxd_tail: 1939 dev.igb.1.queue2.rx_packets: 43909016 dev.igb.1.queue2.rx_bytes: 8673941461 dev.igb.1.queue3.interrupt_rate: 14925 dev.igb.1.queue3.txd_head: 647 dev.igb.1.queue3.txd_tail: 647 dev.igb.1.queue3.tx_packets: 58776199 dev.igb.1.queue3.rxd_head: 692 dev.igb.1.queue3.rxd_tail: 691 dev.igb.1.queue3.rx_packets: 55138996 dev.igb.1.queue3.rx_bytes: 9310217354 dev.igb.1.queue4.interrupt_rate: 100000 dev.igb.1.queue4.txd_head: 1721 dev.igb.1.queue4.txd_tail: 1721 dev.igb.1.queue4.tx_packets: 54337209 dev.igb.1.queue4.rxd_head: 1609 dev.igb.1.queue4.rxd_tail: 1598 dev.igb.1.queue4.rx_packets: 46546503 dev.igb.1.queue4.rx_bytes: 8818182840 dev.igb.1.queue5.interrupt_rate: 11627 dev.igb.1.queue5.txd_head: 254 dev.igb.1.queue5.txd_tail: 254 dev.igb.1.queue5.tx_packets: 53117182 dev.igb.1.queue5.rxd_head: 701 dev.igb.1.queue5.rxd_tail: 685 dev.igb.1.queue5.rx_packets: 43014837 dev.igb.1.queue5.rx_bytes: 8699057447 dev.igb.1.queue6.interrupt_rate: 55555 dev.igb.1.queue6.txd_head: 8 dev.igb.1.queue6.txd_tail: 8 dev.igb.1.queue6.tx_packets: 52654088 dev.igb.1.queue6.rxd_head: 1057 dev.igb.1.queue6.rxd_tail: 1041 dev.igb.1.queue6.rx_packets: 45227030 dev.igb.1.queue6.rx_bytes: 9494489640 dev.igb.1.queue7.interrupt_rate: 5235 dev.igb.1.queue7.txd_head: 729 dev.igb.1.queue7.txd_tail: 729 dev.igb.1.queue7.tx_packets: 61926105 dev.igb.1.queue7.rxd_head: 146 dev.igb.1.queue7.rxd_tail: 140 dev.igb.1.queue7.rx_packets: 51781775 dev.igb.1.queue7.rx_bytes: 8901279226 dev.igb.1.mac_stats.missed_packets: 1657 dev.igb.1.mac_stats.recv_no_buff: 405 dev.igb.1.mac_stats.total_pkts_recvd: 384332760 dev.igb.1.mac_stats.good_pkts_recvd: 384331103 dev.igb.1.mac_stats.bcast_pkts_recvd: 15510 dev.igb.1.mac_stats.mcast_pkts_recvd: 52957 dev.igb.1.mac_stats.rx_frames_64: 195496498 dev.igb.1.mac_stats.rx_frames_65_127: 133346124 dev.igb.1.mac_stats.rx_frames_128_255: 5254911 dev.igb.1.mac_stats.rx_frames_256_511: 9700049 dev.igb.1.mac_stats.rx_frames_512_1023: 16885886 dev.igb.1.mac_stats.rx_frames_1024_1522: 23647635 dev.igb.1.mac_stats.good_octets_recvd: 74284029276 dev.igb.1.mac_stats.good_octets_txd: 544536708502 dev.igb.1.mac_stats.total_pkts_txd: 485327419 dev.igb.1.mac_stats.good_pkts_txd: 485327419 dev.igb.1.mac_stats.bcast_pkts_txd: 72 dev.igb.1.mac_stats.mcast_pkts_txd: 52820 dev.igb.1.mac_stats.tx_frames_64: 57820809 dev.igb.1.mac_stats.tx_frames_65_127: 51586341 dev.igb.1.mac_stats.tx_frames_128_255: 7050579 dev.igb.1.mac_stats.tx_frames_256_511: 7887126 dev.igb.1.mac_stats.tx_frames_512_1023: 10130891 dev.igb.1.mac_stats.tx_frames_1024_1522: 350851673 dev.igb.1.interrupts.asserts: 551135045 dev.igb.1.interrupts.rx_pkt_timer: 384326679 dev.igb.1.interrupts.tx_queue_empty: 485323376 dev.igb.1.interrupts.tx_queue_min_thresh: 6324386 dev.igb.1.host.rx_pkt: 4424 dev.igb.1.host.tx_good_pkt: 4043 dev.igb.1.host.rx_good_bytes: 74284030864 dev.igb.1.host.tx_good_bytes: 544536708502 dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection version - 2.3.1 dev.igb.2.%driver: igb dev.igb.2.%location: slot=0 function=2 dev.igb.2.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x8086 subdevice=0x5001 class=0x020000 dev.igb.2.%parent: pci5 dev.igb.2.nvm: -1 dev.igb.2.enable_aim: 1 dev.igb.2.fc: 3 dev.igb.2.rx_processing_limit: 4096 dev.igb.2.eee_disabled: 1 dev.igb.2.link_irq: 2 dev.igb.2.device_control: 1209795137 dev.igb.2.rx_control: 67141658 dev.igb.2.interrupt_mask: 4 dev.igb.2.extended_int_mask: 2147483959 dev.igb.2.fc_high_water: 33168 dev.igb.2.fc_low_water: 33152 dev.igb.2.queue0.interrupt_rate: 13698 dev.igb.2.queue0.txd_head: 1618 dev.igb.2.queue0.txd_tail: 1618 dev.igb.2.queue0.tx_packets: 46401106 dev.igb.2.queue0.rxd_head: 831 dev.igb.2.queue0.rxd_tail: 827 dev.igb.2.queue0.rx_packets: 69356350 dev.igb.2.queue0.rx_bytes: 68488772907 dev.igb.2.queue1.interrupt_rate: 5405 dev.igb.2.queue1.txd_head: 190 dev.igb.2.queue1.txd_tail: 190 dev.igb.2.queue1.tx_packets: 55965886 dev.igb.2.queue1.rxd_head: 268 dev.igb.2.queue1.rxd_tail: 256 dev.igb.2.queue1.rx_packets: 58958084 dev.igb.2.queue1.rx_bytes: 69154569937 dev.igb.2.queue2.interrupt_rate: 83333 dev.igb.2.queue2.txd_head: 568 dev.igb.2.queue2.txd_tail: 568 dev.igb.2.queue2.tx_packets: 44974648 dev.igb.2.queue2.rxd_head: 371 dev.igb.2.queue2.rxd_tail: 219 dev.igb.2.queue2.rx_packets: 67037407 dev.igb.2.queue2.rx_bytes: 72042326102 dev.igb.2.queue3.interrupt_rate: 12658 dev.igb.2.queue3.txd_head: 867 dev.igb.2.queue3.txd_tail: 867 dev.igb.2.queue3.tx_packets: 55962467 dev.igb.2.queue3.rxd_head: 85 dev.igb.2.queue3.rxd_tail: 1953 dev.igb.2.queue3.rx_packets: 60972965 dev.igb.2.queue3.rx_bytes: 70397176035 dev.igb.2.queue4.interrupt_rate: 90909 dev.igb.2.queue4.txd_head: 1920 dev.igb.2.queue4.txd_tail: 1920 dev.igb.2.queue4.tx_packets: 47660931 dev.igb.2.queue4.rxd_head: 1397 dev.igb.2.queue4.rxd_tail: 1379 dev.igb.2.queue4.rx_packets: 59110758 dev.igb.2.queue4.rx_bytes: 68919201478 dev.igb.2.queue5.interrupt_rate: 111111 dev.igb.2.queue5.txd_head: 886 dev.igb.2.queue5.txd_tail: 886 dev.igb.2.queue5.tx_packets: 45103990 dev.igb.2.queue5.rxd_head: 812 dev.igb.2.queue5.rxd_tail: 799 dev.igb.2.queue5.rx_packets: 59030312 dev.igb.2.queue5.rx_bytes: 69234293962 dev.igb.2.queue6.interrupt_rate: 5208 dev.igb.2.queue6.txd_head: 1926 dev.igb.2.queue6.txd_tail: 1926 dev.igb.2.queue6.tx_packets: 46215046 dev.igb.2.queue6.rxd_head: 692 dev.igb.2.queue6.rxd_tail: 689 dev.igb.2.queue6.rx_packets: 58256050 dev.igb.2.queue6.rx_bytes: 68429172749 dev.igb.2.queue7.interrupt_rate: 50000 dev.igb.2.queue7.txd_head: 126 dev.igb.2.queue7.txd_tail: 126 dev.igb.2.queue7.tx_packets: 52451455 dev.igb.2.queue7.rxd_head: 968 dev.igb.2.queue7.rxd_tail: 885 dev.igb.2.queue7.rx_packets: 65946491 dev.igb.2.queue7.rx_bytes: 70263478849 dev.igb.2.mac_stats.missed_packets: 958 dev.igb.2.mac_stats.recv_no_buff: 69 dev.igb.2.mac_stats.total_pkts_recvd: 498658079 dev.igb.2.mac_stats.good_pkts_recvd: 498657121 dev.igb.2.mac_stats.bcast_pkts_recvd: 16867 dev.igb.2.mac_stats.mcast_pkts_recvd: 52957 dev.igb.2.mac_stats.rx_frames_64: 59089332 dev.igb.2.mac_stats.rx_frames_65_127: 52880118 dev.igb.2.mac_stats.rx_frames_128_255: 7526966 dev.igb.2.mac_stats.rx_frames_256_511: 8468389 dev.igb.2.mac_stats.rx_frames_512_1023: 10434770 dev.igb.2.mac_stats.rx_frames_1024_1522: 360257545 dev.igb.2.mac_stats.good_octets_recvd: 558910494322 dev.igb.2.mac_stats.good_octets_txd: 84618858153 dev.igb.2.mac_stats.total_pkts_txd: 394726904 dev.igb.2.mac_stats.good_pkts_txd: 394726904 dev.igb.2.mac_stats.bcast_pkts_txd: 48 dev.igb.2.mac_stats.mcast_pkts_txd: 52821 dev.igb.2.mac_stats.tx_frames_64: 196605932 dev.igb.2.mac_stats.tx_frames_65_127: 134602807 dev.igb.2.mac_stats.tx_frames_128_255: 5705236 dev.igb.2.mac_stats.tx_frames_256_511: 10267168 dev.igb.2.mac_stats.tx_frames_512_1023: 17165496 dev.igb.2.mac_stats.tx_frames_1024_1522: 30380265 dev.igb.2.interrupts.asserts: 465994260 dev.igb.2.interrupts.rx_pkt_timer: 498647546 dev.igb.2.interrupts.tx_queue_empty: 394720657 dev.igb.2.interrupts.tx_queue_min_thresh: 24424555 dev.igb.2.host.rx_pkt: 9575 dev.igb.2.host.tx_good_pkt: 6248 dev.igb.2.host.rx_good_bytes: 558910513984 dev.igb.2.host.tx_good_bytes: 84618858217 Thanks for your help. Cheers, -- Clement (nodens)