Date: Sun, 28 Apr 2013 07:21:42 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: freebsd-net@freebsd.org, =?iso-8859-1?Q?Cl=E9ment_Hermann_=28nodens=29?= <nodens2099@gmail.com> Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 Message-ID: <1367158902.96158.YahooMailClassic@web121603.mail.ne1.yahoo.com> In-Reply-To: <517A657B.7060003@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
=0A=0A--- On Fri, 4/26/13, "Cl=E9ment Hermann (nodens)" <nodens2099@gmail.c= om> wrote:=0A=0A> From: "Cl=E9ment Hermann (nodens)" <nodens2099@gmail.com>= =0A> Subject: High CPU interrupt load on intel I350T4 with igb on 8.3=0A> T= o: freebsd-net@freebsd.org=0A> Date: Friday, April 26, 2013, 7:31 AM=0A> Hi= list,=0A> =0A> We use pf+ALTQ for trafic shaping on some routers.=0A> =0A>= We are switching to new servers : Dell PowerEdge R620 with 2=0A> 8-cores I= ntel Processor (E5-2650L), 8GB RAM and Intel I350T4=0A> (quad port) using i= gb driver. The old hardware is using em=0A> driver, the CPU load is high bu= t mostly due to kernel and a=0A> large pf ruleset.=0A> =0A> On the new hard= ware, we see high CPU Interrupt load (up to=0A> 95%), even though there is = not much trafic currently (peaks=0A> about 150Mbps and 40Kpps). All queues = are used and binded to=0A> a cpu according to top, but a lot of CPU time is= spent on=0A> igb queues (interrupt or wait). The load is fine when we=0A> = stay below 20Kpps.=0A> =0A> We see no mbuf shortage, no dropped packet, but= there is=0A> little margin left on CPU time (about 25% idle at best, most= =0A> of CPU time is spent on interrupts), which is disturbing.=0A> =0A> We = have done some tuning, but to no avail :=0A> =0A> sysctl.conf :=0A> =0A> # = mbufs=0A> kern.ipc.nmbclusters=3D65536=0A> # Sockets=0A> kern.ipc.somaxconn= =3D8192=0A> net.inet.tcp.delayed_ack=3D0=0A> net.inet.tcp.sendspace=3D65535= =0A> net.inet.udp.recvspace=3D65535=0A> net.inet.udp.maxdgram=3D57344=0A> n= et.local.stream.recvspace=3D65535=0A> net.local.stream.sendspace=3D65535=0A= > # IGB=0A> dev.igb.0.rx_processing_limit=3D4096=0A> dev.igb.1.rx_processin= g_limit=3D4096=0A> dev.igb.2.rx_processing_limit=3D4096=0A> dev.igb.3.rx_pr= ocessing_limit=3D4096=0A> =0A> /boot/loader.conf :=0A> =0A> vm.kmem_size=3D= 1G=0A> hw.igb.max_interrupt_rate=3D"32000"=A0 # maximum number of=0A> inter= rupts/sec generated by single igb(4) (default 8000)=0A> hw.igb.txd=3D"2048"= =A0 =A0 =A0 =A0 =A0 =A0=0A> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=0A> =A0 # numbe= r of transmit descriptors allocated by the=0A> driver (2048 limit)=0A> hw.i= gb.rxd=3D"2048"=A0 =A0 =A0 =A0 =A0 =A0=0A> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =0A> =A0 # number of receive descriptors allocated by the=0A> driver (2048 = limit)=0A> hw.igb.rx_process_limit=3D"1000"=A0 =A0=A0=A0#=0A> maximum numbe= r of received packets to process at a time, The=0A> default of 100 is=0A> = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=0A> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=0A> =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0=0A> =A0 =A0 =A0 =A0 =A0 =A0=0A> =A0=A0=A0# too= low for most firewalls. (-1 means=0A> unlimited)=0A> =0A> Kernel HZ is 100= 0.=0A> =0A> The IGB /boot/loader.conf tuning was our last attempt, it=0A> d= idn't change anything.=0A> =0A> Does anyone have any pointer ? How could we= lower CPU=0A> interrupt load ? should we set hw.igb.max_interrupt_rate=0A>= lower instead of higher ?=0A> From what we saw here and there, we should b= e able to do=0A> much better with this hardware.=0A> =0A> =0A> relevant sys= ctl (igb1 and igb2 only, other interfaces are=0A> unused) :=0A> =0A> sysctl= dev.igb | grep -v ": 0$"=0A> dev.igb.1.%desc: Intel(R) PRO/1000 Network Co= nnection=0A> version - 2.3.1=0A> dev.igb.1.%driver: igb=0A> dev.igb.1.%loca= tion: slot=3D0 function=3D1=0A> dev.igb.1.%pnpinfo: vendor=3D0x8086 device= =3D0x1521=0A> subvendor=3D0x8086 subdevice=3D0x5001 class=3D0x020000=0A> de= v.igb.1.%parent: pci5=0A> dev.igb.1.nvm: -1=0A> dev.igb.1.enable_aim: 1=0A>= dev.igb.1.fc: 3=0A> dev.igb.1.rx_processing_limit: 4096=0A> dev.igb.1.eee_= disabled: 1=0A> dev.igb.1.link_irq: 2=0A> dev.igb.1.device_control: 1209795= 137=0A> dev.igb.1.rx_control: 67141658=0A> dev.igb.1.interrupt_mask: 4=0A> = dev.igb.1.extended_int_mask: 2147483981=0A> dev.igb.1.fc_high_water: 33168= =0A> dev.igb.1.fc_low_water: 33152=0A> dev.igb.1.queue0.interrupt_rate: 714= 28=0A> dev.igb.1.queue0.txd_head: 1318=0A> dev.igb.1.queue0.txd_tail: 1318= =0A> dev.igb.1.queue0.tx_packets: 84663594=0A> dev.igb.1.queue0.rxd_head: 7= 17=0A> dev.igb.1.queue0.rxd_tail: 715=0A> dev.igb.1.queue0.rx_packets: 4389= 9597=0A> dev.igb.1.queue0.rx_bytes: 8905556030=0A> dev.igb.1.queue1.interru= pt_rate: 90909=0A> dev.igb.1.queue1.txd_head: 693=0A> dev.igb.1.queue1.txd_= tail: 693=0A> dev.igb.1.queue1.tx_packets: 57543349=0A> dev.igb.1.queue1.rx= d_head: 1033=0A> dev.igb.1.queue1.rxd_tail: 1032=0A> dev.igb.1.queue1.rx_pa= ckets: 54821897=0A> dev.igb.1.queue1.rx_bytes: 9944955108=0A> dev.igb.1.que= ue2.interrupt_rate: 100000=0A> dev.igb.1.queue2.txd_head: 350=0A> dev.igb.1= .queue2.txd_tail: 350=0A> dev.igb.1.queue2.tx_packets: 62320990=0A> dev.igb= .1.queue2.rxd_head: 1962=0A> dev.igb.1.queue2.rxd_tail: 1939=0A> dev.igb.1.= queue2.rx_packets: 43909016=0A> dev.igb.1.queue2.rx_bytes: 8673941461=0A> d= ev.igb.1.queue3.interrupt_rate: 14925=0A> dev.igb.1.queue3.txd_head: 647=0A= > dev.igb.1.queue3.txd_tail: 647=0A> dev.igb.1.queue3.tx_packets: 58776199= =0A> dev.igb.1.queue3.rxd_head: 692=0A> dev.igb.1.queue3.rxd_tail: 691=0A> = dev.igb.1.queue3.rx_packets: 55138996=0A> dev.igb.1.queue3.rx_bytes: 931021= 7354=0A> dev.igb.1.queue4.interrupt_rate: 100000=0A> dev.igb.1.queue4.txd_h= ead: 1721=0A> dev.igb.1.queue4.txd_tail: 1721=0A> dev.igb.1.queue4.tx_packe= ts: 54337209=0A> dev.igb.1.queue4.rxd_head: 1609=0A> dev.igb.1.queue4.rxd_t= ail: 1598=0A> dev.igb.1.queue4.rx_packets: 46546503=0A> dev.igb.1.queue4.rx= _bytes: 8818182840=0A> dev.igb.1.queue5.interrupt_rate: 11627=0A> dev.igb.1= .queue5.txd_head: 254=0A> dev.igb.1.queue5.txd_tail: 254=0A> dev.igb.1.queu= e5.tx_packets: 53117182=0A> dev.igb.1.queue5.rxd_head: 701=0A> dev.igb.1.qu= eue5.rxd_tail: 685=0A> dev.igb.1.queue5.rx_packets: 43014837=0A> dev.igb.1.= queue5.rx_bytes: 8699057447=0A> dev.igb.1.queue6.interrupt_rate: 55555=0A> = dev.igb.1.queue6.txd_head: 8=0A> dev.igb.1.queue6.txd_tail: 8=0A> dev.igb.1= .queue6.tx_packets: 52654088=0A> dev.igb.1.queue6.rxd_head: 1057=0A> dev.ig= b.1.queue6.rxd_tail: 1041=0A> dev.igb.1.queue6.rx_packets: 45227030=0A> dev= .igb.1.queue6.rx_bytes: 9494489640=0A> dev.igb.1.queue7.interrupt_rate: 523= 5=0A> dev.igb.1.queue7.txd_head: 729=0A> dev.igb.1.queue7.txd_tail: 729=0A>= dev.igb.1.queue7.tx_packets: 61926105=0A> dev.igb.1.queue7.rxd_head: 146= =0A> dev.igb.1.queue7.rxd_tail: 140=0A> dev.igb.1.queue7.rx_packets: 517817= 75=0A> dev.igb.1.queue7.rx_bytes: 8901279226=0A> dev.igb.1.mac_stats.missed= _packets: 1657=0A> dev.igb.1.mac_stats.recv_no_buff: 405=0A> dev.igb.1.mac_= stats.total_pkts_recvd: 384332760=0A> dev.igb.1.mac_stats.good_pkts_recvd: = 384331103=0A> dev.igb.1.mac_stats.bcast_pkts_recvd: 15510=0A> dev.igb.1.mac= _stats.mcast_pkts_recvd: 52957=0A> dev.igb.1.mac_stats.rx_frames_64: 195496= 498=0A> dev.igb.1.mac_stats.rx_frames_65_127: 133346124=0A> dev.igb.1.mac_s= tats.rx_frames_128_255: 5254911=0A> dev.igb.1.mac_stats.rx_frames_256_511: = 9700049=0A> dev.igb.1.mac_stats.rx_frames_512_1023: 16885886=0A> dev.igb.1.= mac_stats.rx_frames_1024_1522: 23647635=0A> dev.igb.1.mac_stats.good_octets= _recvd: 74284029276=0A> dev.igb.1.mac_stats.good_octets_txd: 544536708502= =0A> dev.igb.1.mac_stats.total_pkts_txd: 485327419=0A> dev.igb.1.mac_stats.= good_pkts_txd: 485327419=0A> dev.igb.1.mac_stats.bcast_pkts_txd: 72=0A> dev= .igb.1.mac_stats.mcast_pkts_txd: 52820=0A> dev.igb.1.mac_stats.tx_frames_64= : 57820809=0A> dev.igb.1.mac_stats.tx_frames_65_127: 51586341=0A> dev.igb.1= .mac_stats.tx_frames_128_255: 7050579=0A> dev.igb.1.mac_stats.tx_frames_256= _511: 7887126=0A> dev.igb.1.mac_stats.tx_frames_512_1023: 10130891=0A> dev.= igb.1.mac_stats.tx_frames_1024_1522: 350851673=0A> dev.igb.1.interrupts.ass= erts: 551135045=0A> dev.igb.1.interrupts.rx_pkt_timer: 384326679=0A> dev.ig= b.1.interrupts.tx_queue_empty: 485323376=0A> dev.igb.1.interrupts.tx_queue_= min_thresh: 6324386=0A> dev.igb.1.host.rx_pkt: 4424=0A> dev.igb.1.host.tx_g= ood_pkt: 4043=0A> dev.igb.1.host.rx_good_bytes: 74284030864=0A> dev.igb.1.h= ost.tx_good_bytes: 544536708502=0A> dev.igb.2.%desc: Intel(R) PRO/1000 Netw= ork Connection=0A> version - 2.3.1=0A> dev.igb.2.%driver: igb=0A> dev.igb.2= .%location: slot=3D0 function=3D2=0A> dev.igb.2.%pnpinfo: vendor=3D0x8086 d= evice=3D0x1521=0A> subvendor=3D0x8086 subdevice=3D0x5001 class=3D0x020000= =0A> dev.igb.2.%parent: pci5=0A> dev.igb.2.nvm: -1=0A> dev.igb.2.enable_aim= : 1=0A> dev.igb.2.fc: 3=0A> dev.igb.2.rx_processing_limit: 4096=0A> dev.igb= .2.eee_disabled: 1=0A> dev.igb.2.link_irq: 2=0A> dev.igb.2.device_control: = 1209795137=0A> dev.igb.2.rx_control: 67141658=0A> dev.igb.2.interrupt_mask:= 4=0A> dev.igb.2.extended_int_mask: 2147483959=0A> dev.igb.2.fc_high_water:= 33168=0A> dev.igb.2.fc_low_water: 33152=0A> dev.igb.2.queue0.interrupt_rat= e: 13698=0A> dev.igb.2.queue0.txd_head: 1618=0A> dev.igb.2.queue0.txd_tail:= 1618=0A> dev.igb.2.queue0.tx_packets: 46401106=0A> dev.igb.2.queue0.rxd_he= ad: 831=0A> dev.igb.2.queue0.rxd_tail: 827=0A> dev.igb.2.queue0.rx_packets:= 69356350=0A> dev.igb.2.queue0.rx_bytes: 68488772907=0A> dev.igb.2.queue1.i= nterrupt_rate: 5405=0A> dev.igb.2.queue1.txd_head: 190=0A> dev.igb.2.queue1= .txd_tail: 190=0A> dev.igb.2.queue1.tx_packets: 55965886=0A> dev.igb.2.queu= e1.rxd_head: 268=0A> dev.igb.2.queue1.rxd_tail: 256=0A> dev.igb.2.queue1.rx= _packets: 58958084=0A> dev.igb.2.queue1.rx_bytes: 69154569937=0A> dev.igb.2= .queue2.interrupt_rate: 83333=0A> dev.igb.2.queue2.txd_head: 568=0A> dev.ig= b.2.queue2.txd_tail: 568=0A> dev.igb.2.queue2.tx_packets: 44974648=0A> dev.= igb.2.queue2.rxd_head: 371=0A> dev.igb.2.queue2.rxd_tail: 219=0A> dev.igb.2= .queue2.rx_packets: 67037407=0A> dev.igb.2.queue2.rx_bytes: 72042326102=0A>= dev.igb.2.queue3.interrupt_rate: 12658=0A> dev.igb.2.queue3.txd_head: 867= =0A> dev.igb.2.queue3.txd_tail: 867=0A> dev.igb.2.queue3.tx_packets: 559624= 67=0A> dev.igb.2.queue3.rxd_head: 85=0A> dev.igb.2.queue3.rxd_tail: 1953=0A= > dev.igb.2.queue3.rx_packets: 60972965=0A> dev.igb.2.queue3.rx_bytes: 7039= 7176035=0A> dev.igb.2.queue4.interrupt_rate: 90909=0A> dev.igb.2.queue4.txd= _head: 1920=0A> dev.igb.2.queue4.txd_tail: 1920=0A> dev.igb.2.queue4.tx_pac= kets: 47660931=0A> dev.igb.2.queue4.rxd_head: 1397=0A> dev.igb.2.queue4.rxd= _tail: 1379=0A> dev.igb.2.queue4.rx_packets: 59110758=0A> dev.igb.2.queue4.= rx_bytes: 68919201478=0A> dev.igb.2.queue5.interrupt_rate: 111111=0A> dev.i= gb.2.queue5.txd_head: 886=0A> dev.igb.2.queue5.txd_tail: 886=0A> dev.igb.2.= queue5.tx_packets: 45103990=0A> dev.igb.2.queue5.rxd_head: 812=0A> dev.igb.= 2.queue5.rxd_tail: 799=0A> dev.igb.2.queue5.rx_packets: 59030312=0A> dev.ig= b.2.queue5.rx_bytes: 69234293962=0A> dev.igb.2.queue6.interrupt_rate: 5208= =0A> dev.igb.2.queue6.txd_head: 1926=0A> dev.igb.2.queue6.txd_tail: 1926=0A= > dev.igb.2.queue6.tx_packets: 46215046=0A> dev.igb.2.queue6.rxd_head: 692= =0A> dev.igb.2.queue6.rxd_tail: 689=0A> dev.igb.2.queue6.rx_packets: 582560= 50=0A> dev.igb.2.queue6.rx_bytes: 68429172749=0A> dev.igb.2.queue7.interrup= t_rate: 50000=0A> dev.igb.2.queue7.txd_head: 126=0A> dev.igb.2.queue7.txd_t= ail: 126=0A> dev.igb.2.queue7.tx_packets: 52451455=0A> dev.igb.2.queue7.rxd= _head: 968=0A> dev.igb.2.queue7.rxd_tail: 885=0A> dev.igb.2.queue7.rx_packe= ts: 65946491=0A> dev.igb.2.queue7.rx_bytes: 70263478849=0A> dev.igb.2.mac_s= tats.missed_packets: 958=0A> dev.igb.2.mac_stats.recv_no_buff: 69=0A> dev.i= gb.2.mac_stats.total_pkts_recvd: 498658079=0A> dev.igb.2.mac_stats.good_pkt= s_recvd: 498657121=0A> dev.igb.2.mac_stats.bcast_pkts_recvd: 16867=0A> dev.= igb.2.mac_stats.mcast_pkts_recvd: 52957=0A> dev.igb.2.mac_stats.rx_frames_6= 4: 59089332=0A> dev.igb.2.mac_stats.rx_frames_65_127: 52880118=0A> dev.igb.= 2.mac_stats.rx_frames_128_255: 7526966=0A> dev.igb.2.mac_stats.rx_frames_25= 6_511: 8468389=0A> dev.igb.2.mac_stats.rx_frames_512_1023: 10434770=0A> dev= .igb.2.mac_stats.rx_frames_1024_1522: 360257545=0A> dev.igb.2.mac_stats.goo= d_octets_recvd: 558910494322=0A> dev.igb.2.mac_stats.good_octets_txd: 84618= 858153=0A> dev.igb.2.mac_stats.total_pkts_txd: 394726904=0A> dev.igb.2.mac_= stats.good_pkts_txd: 394726904=0A> dev.igb.2.mac_stats.bcast_pkts_txd: 48= =0A> dev.igb.2.mac_stats.mcast_pkts_txd: 52821=0A> dev.igb.2.mac_stats.tx_f= rames_64: 196605932=0A> dev.igb.2.mac_stats.tx_frames_65_127: 134602807=0A>= dev.igb.2.mac_stats.tx_frames_128_255: 5705236=0A> dev.igb.2.mac_stats.tx_= frames_256_511: 10267168=0A> dev.igb.2.mac_stats.tx_frames_512_1023: 171654= 96=0A> dev.igb.2.mac_stats.tx_frames_1024_1522: 30380265=0A> dev.igb.2.inte= rrupts.asserts: 465994260=0A> dev.igb.2.interrupts.rx_pkt_timer: 498647546= =0A> dev.igb.2.interrupts.tx_queue_empty: 394720657=0A> dev.igb.2.interrupt= s.tx_queue_min_thresh: 24424555=0A> dev.igb.2.host.rx_pkt: 9575=0A> dev.igb= .2.host.tx_good_pkt: 6248=0A> dev.igb.2.host.rx_good_bytes: 558910513984=0A= > dev.igb.2.host.tx_good_bytes: 84618858217=0A> =0A> =0A> Thanks for your h= elp.=0A> =0A> Cheers,=0A=0AYou're experiencing lock contention=0A=0ATry edi= ting igb.c and setting the queues to 1. The multiqueue=0Aimplementation in = igb has a negative impact.=0A=0AIf you have just 1 system get a dual port 8= 2571 card and use the=0Aem driver.=0A=0ABC
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1367158902.96158.YahooMailClassic>