Date: Mon, 24 Oct 2011 09:59:43 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Sergey Saley <sergeysaley@gmail.com> Cc: freebsd-net@freebsd.org Subject: Re: Too much interrupts on ixgbe Message-ID: <CAFOYbc=vM6_UMXCTOUz_mTyTVpCxp61Np68ERH8tx4KE9aT8gA@mail.gmail.com> In-Reply-To: <1319449307149-4931883.post@n5.nabble.com> References: <1319449307149-4931883.post@n5.nabble.com>
next in thread | previous in thread | raw e-mail | index | archive | help
You need to increase your mbuf pool, note below in your messages where it has insufficient to configure for ix1, I don't know that this will change the interrupt difference but it should be addressed anyway. I also noticed that you have the adapter in a PCIE 1.0 slot (only 2.5 Gb/s), that will limit you, you should move to a 2.0 slot if possible. Regards, Jack On Mon, Oct 24, 2011 at 2:41 AM, Sergey Saley <sergeysaley@gmail.com> wrote: > There is my FreeBSD box: > > kernel > --------------- > # > # GENERIC -- Generic kernel configuration file for FreeBSD/i386 > # > # For more information on this file, please read the config(5) manual page, > # and/or the handbook section on Kernel Configuration Files: > # > # > > http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html > # > # The handbook is also available locally in /usr/share/doc/handbook > # if you've installed the doc distribution, otherwise always see the > # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the > # latest information. > # > # An exhaustive list of options and more detailed explanations of the > # device lines is also present in the ../../conf/NOTES and NOTES files. > # If you are in doubt as to the purpose or necessity of a line, check first > # in NOTES. > # > # $FreeBSD: head/sys/i386/conf/GENERIC 221743 2011-05-10 16:44:16Z jkim $ > > cpu I686_CPU > ident POINT07 > > > options SCHED_ULE # ULE scheduler > options PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates support > options UFS_DIRHASH # Improve performance on big > directories > options MD_ROOT # MD is a potential root device > options MSDOSFS # MSDOS Filesystem > options CD9660 # ISO 9660 Filesystem > options PROCFS # Process filesystem (requires > PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options KTRACE # ktrace(1) support > options STACK # stack(9) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time > extensions > options PRINTF_BUFR_SIZE=128 # Prevent printf output being > interspersed. > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options HWPMC_HOOKS # Necessary kernel hooks for > hwpmc(4) > #options KDTRACE_HOOKS # Kernel DTrace hooks > options INCLUDE_CONFIG_FILE # Include this file in kernel > > > # To make an SMP kernel, the next two lines are needed > options SMP # Symmetric MultiProcessor Kernel > device apic # I/O APIC > > # CPU frequency control > device cpufreq > > # Bus support. > device acpi > device pci > > # Floppy drives > device fdc > > # ATA controllers > device ahci # AHCI-compatible SATA controllers > device ata # Legacy ATA/SATA controllers > options ATA_CAM # Handle legacy controllers with CAM > options ATA_STATIC_ID # Static device numbering > device mvs # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC > SATA > device siis # SiliconImage SiI3124/SiI3132/SiI3531 SATA > > # ATA/SCSI peripherals > device scbus # SCSI bus (required for ATA/SCSI) > device ch # SCSI media changers > device da # Direct Access (disks) > device sa # Sequential Access (tape etc) > device cd # CD > device pass # Passthrough device (direct ATA/SCSI > access) > device ses # SCSI Environmental Services (and SAF-TE) > > > # atkbdc0 controls both the keyboard and the PS/2 mouse > device atkbdc # AT keyboard controller > device atkbd # AT keyboard > device psm # PS/2 mouse > > device kbdmux # keyboard multiplexer > > device vga # VGA video card driver > > > # syscons is the default console driver, resembling an SCO console > device sc > options SC_PIXEL_MODE # add support for the raster text mode > > device agp # support several AGP chipsets > > # Power management support (see NOTES for more options) > #device apm > # Add suspend/resume support for the i8254. > device pmtimer > > # Serial (COM) ports > device uart # Generic UART driver > > # Parallel port > device ppc > device ppbus # Parallel port bus (required) > device lpt # Printer > #device plip # TCP/IP over parallel > device ppi # Parallel port interface device > #device vpo # Requires scbus and da > > # If you've got a "dumb" serial or parallel PCI card that is > # supported by the puc(4) glue driver, uncomment the following > # line to enable it (connects to sio, uart and/or ppc drivers): > #device puc > > # PCI Ethernet NICs. > device em # Intel PRO/1000 Gigabit Ethernet Family > device ixgbe # Intel PRO/10GbE Ethernet Card > > # Pseudo devices. > device loop # Network loopback > device random # Entropy device > device ether # Ethernet support > device vlan # 802.1Q VLAN support > #device tun # Packet tunnel. > device pty # BSD-style compatibility pseudo ttys > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device faith # IPv6-to-IPv4 relaying (translation) > device firmware # firmware assist module > > # The `bpf' device enables the Berkeley Packet Filter. > # Be aware of the administrative consequences of enabling this! > # Note that 'bpf' is required for DHCP. > device bpf # Berkeley packet filter > > > options IPFIREWALL > options IPFIREWALL_DEFAULT_TO_ACCEPT > options IPFIREWALL_FORWARD > options DUMMYNET > options IPI_PREEMPTION > > # netgraph(4). Enable the base netgraph code with the NETGRAPH option. > # Individual node types can be enabled with the corresponding option > # listed below; however, this is not strictly necessary as netgraph > # will automatically load the corresponding KLD module if the node type > # is not already compiled into the kernel. Each type below has a > # corresponding man page, e.g., ng_async(8). > > options NETGRAPH # netgraph(4) system > > options NETGRAPH_CAR > options NETGRAPH_IFACE > options NETGRAPH_MPPC_ENCRYPTION > options NETGRAPH_PPP > options NETGRAPH_PPPOE > options NETGRAPH_PPTPGRE > options NETGRAPH_SOCKET > options NETGRAPH_TCPMSS > options NETGRAPH_ETHER > options NETGRAPH_TEE > options NETGRAPH_VJC > options NETGRAPH_BPF > options NETGRAPH_KSOCKET > options NETGRAPH_IPFW > > dmesg.boot > -------------------------- > Copyright (c) 1992-2011 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-BETA3 #1: Fri Oct 21 10:37:43 FET 2011 > root@point07.uch.net:/usr/obj/usr/src/sys/POINT07 amd64 > CPU: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz (2394.05-MHz K8-class > CPU) > Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 > > > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM> > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x1<LAHF> > TSC: P-state invariant, performance statistics > real memory = 2147483648 (2048 MB) > avail memory = 2056187904 (1960 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: <PTLTD APIC > > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > FreeBSD/SMP: 1 package(s) x 4 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > ioapic0 <Version 2.0> irqs 0-23 on motherboard > kbd1 at kbdmux0 > acpi0: <PTLTD XSDT> on motherboard > acpi0: Power Button (fixed) > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 > cpu0: <ACPI CPU> on acpi0 > cpu1: <ACPI CPU> on acpi0 > cpu2: <ACPI CPU> on acpi0 > cpu3: <ACPI CPU> on acpi0 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0 > pci1: <ACPI PCI bus> on pcib1 > ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11> port > 0x2000-0x201f mem 0xdc680000-0xdc6fffff,0xdc600000-0xdc603fff irq 16 at > device 0.0 on pci1 > ix0: Using MSIX interrupts with 5 vectors > ix0: Ethernet address: 00:25:90:3c:42:6a > ix0: PCI Express Bus: Speed 2.5Gb/s Width x8 > ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.11> port > 0x2020-0x203f mem 0xdc700000-0xdc77ffff,0xdc604000-0xdc607fff irq 17 at > device 0.1 on pci1 > ix1: Using MSIX interrupts with 5 vectors > ix1: RX Descriptors exceed system mbuf max, using default instead! > ix1: Ethernet address: 00:25:90:3c:42:6b > ix1: PCI Express Bus: Speed 2.5Gb/s Width x8 > pci0: <serial bus, USB> at device 26.0 (no driver attached) > pci0: <serial bus, USB> at device 26.1 (no driver attached) > pci0: <serial bus, USB> at device 26.2 (no driver attached) > pci0: <serial bus, USB> at device 26.7 (no driver attached) > pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 > pci5: <ACPI PCI bus> on pcib2 > pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0 > pci13: <ACPI PCI bus> on pcib3 > em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x3000-0x301f mem > 0xdc100000-0xdc11ffff irq 16 at device 0.0 on pci13 > em0: Using an MSI interrupt > em0: Ethernet address: 00:30:48:d2:12:ac > pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0 > pci15: <ACPI PCI bus> on pcib4 > em1: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x4000-0x401f mem > 0xdc200000-0xdc21ffff irq 17 at device 0.0 on pci15 > em1: Using an MSI interrupt > em1: Ethernet address: 00:30:48:d2:12:ad > pci0: <serial bus, USB> at device 29.0 (no driver attached) > pci0: <serial bus, USB> at device 29.1 (no driver attached) > pci0: <serial bus, USB> at device 29.2 (no driver attached) > pci0: <serial bus, USB> at device 29.7 (no driver attached) > pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0 > pci17: <ACPI PCI bus> on pcib5 > vgapci0: <VGA-compatible display> port 0x5000-0x507f mem > 0xde000000-0xdfffffff,0xdc300000-0xdc33ffff at device 4.0 on pci17 > isab0: <PCI-ISA bridge> at device 31.0 on pci0 > isa0: <ISA bus> on isab0 > atapci0: <Intel ICH9 SATA300 controller> port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1c10-0x1c1f,0x1c00-0x1c0f at device > 31.2 on pci0 > ata0: <ATA channel 0> on atapci0 > ata1: <ATA channel 1> on atapci0 > pci0: <serial bus, SMBus> at device 31.3 (no driver attached) > atapci1: <Intel ICH9 SATA300 controller> port > > 0x1c68-0x1c6f,0x1c5c-0x1c5f,0x1c60-0x1c67,0x1c58-0x1c5b,0x1c30-0x1c3f,0x1c20-0x1c2f > irq 18 at device 31.5 on pci0 > ata2: <ATA channel 0> on atapci1 > ata3: <ATA channel 1> on atapci1 > pci0: <dasp> at device 31.6 (no driver attached) > acpi_button0: <Power Button> on acpi0 > atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 > atkbd0: <AT Keyboard> irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 > fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 > ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode > ppc0: FIFO with 16/16/9 bytes threshold > ppbus0: <Parallel port bus> on ppc0 > lpt0: <Printer> on ppbus0 > lpt0: Interrupt-driven port > ppi0: <Parallel I/O> on ppbus0 > sc0: <System console> at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > est0: <Enhanced SpeedStep Frequency Control> on cpu0 > p4tcc0: <CPU Frequency Thermal Control> on cpu0 > est1: <Enhanced SpeedStep Frequency Control> on cpu1 > p4tcc1: <CPU Frequency Thermal Control> on cpu1 > est2: <Enhanced SpeedStep Frequency Control> on cpu2 > p4tcc2: <CPU Frequency Thermal Control> on cpu2 > est3: <Enhanced SpeedStep Frequency Control> on cpu3 > p4tcc3: <CPU Frequency Thermal Control> on cpu3 > Timecounters tick every 1.000 msec > ipfw2 initialized, divert loadable, nat loadable, rule-based forwarding > enabled, default to accept, logging disabled > DUMMYNET 0 with IPv6 initialized (100409) > load_dn_sched dn_sched QFQ loaded > load_dn_sched dn_sched RR loaded > load_dn_sched dn_sched WF2Q+ loaded > load_dn_sched dn_sched FIFO loaded > load_dn_sched dn_sched PRIO loaded > ada0 at ata0 bus 0 scbus0 target 0 lun 0 > ada0: <SAMSUNG HD161HJ GF100-07> ATA-8 SATA 2.x device > ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes) > ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C) > ada0: Previously was known as ad0 > SMP: AP CPU #3 Launched! > SMP: AP CPU #1 Launched! > SMP: AP CPU #2 Launched! > Timecounter "TSC-low" frequency 9351750 Hz quality 1000 > Trying to mount root from ufs:/dev/ada0p2 [rw]... > > sysctl.conf > -------------- > net.inet.ip.fastforwarding=1 > net.isr.direct=0 > net.isr.direct_force=0 > net.inet.ip.fw.one_pass=1 > hw.intr_storm_threshold=9000 > kern.ipc.nmbclusters=262144 > kern.ipc.nmbjumbop=262144 > dev.ix.0.rx_processing_limit=4096 > dev.ix.1.rx_processing_limit=4096 > net.inet.ip.intr_queue_maxlen=256 > dev.ix.0.flow_control=0 > dev.ix.1.flow_control=0 > > vmstat -i > ------------- > # vmstat -i > interrupt total rate > irq1: atkbd0 1 0 > irq6: fdc0 1 0 > irq14: ata0 80855 9 > irq15: ata1 1 0 > cpu0:timer 9694291 1122 > irq256: ix0:que 0 54169187 6273 > irq257: ix0:que 1 19307922 2236 > irq258: ix0:que 2 24265573 2810 > irq259: ix0:que 3 25845865 2993 > irq260: ix0:link 2 0 > irq261: ix1:que 0 63210220 7321 > irq262: ix1:que 1 265895 30 > irq263: ix1:que 2 42114 4 > irq264: ix1:que 3 98953 11 > irq265: ix1:link 5 0 > cpu3:timer 7165968 829 > cpu1:timer 7386880 855 > cpu2:timer 7307920 846 > Total 218841653 25346 > > netstat > ------------ > # netstat -I ix0 -w1 > input (ix0) output > packets errs idrops bytes packets errs bytes colls > 26592 0 0 30220498 20757 0 11490639 0 > 31232 0 0 34001150 24699 0 14554631 0 > 27785 0 0 29672129 23699 0 16154542 0 > 25525 0 0 27450183 22868 0 16180216 0 > ^C > # netstat -I ix1 -w1 > input (ix1) output > packets errs idrops bytes packets errs bytes colls > 22398 0 0 17232635 23781 0 24718205 0 > 23671 0 0 17608589 25859 0 28970777 0 > 24096 0 0 17278557 27188 0 30804019 0 > 22043 0 0 16271250 24116 0 26238423 0 > ^C > > > top -SCHP > ----------------- > last pid: 74628; load averages: 1.26, 1.01, 0.89 > up 0+02:25:14 13:22:11 > 110 processes: 9 running, 79 sleeping, 22 waiting > CPU 0: 0.4% user, 0.0% nice, 3.1% system, 44.1% interrupt, 52.4% idle > CPU 1: 0.0% user, 0.0% nice, 3.1% system, 13.4% interrupt, 83.5% idle > CPU 2: 0.8% user, 0.0% nice, 2.4% system, 18.5% interrupt, 78.3% idle > CPU 3: 0.4% user, 0.0% nice, 4.7% system, 13.0% interrupt, 81.9% idle > Mem: 44M Active, 118M Inact, 326M Wired, 792K Cache, 213M Buf, 1487M Free > Swap: 4096M Total, 4096M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND > 10 root 155 ki31 0K 64K RUN 2 130:32 94.68% idle{idle: > cpu2} > 10 root 155 ki31 0K 64K RUN 1 130:49 93.07% idle{idle: > cpu1} > 10 root 155 ki31 0K 64K CPU3 3 132:48 89.26% idle{idle: > cpu3} > 10 root 155 ki31 0K 64K RUN 0 101:00 58.89% idle{idle: > cpu0} > 11 root -92 - 0K 416K CPU0 0 30:52 36.77% intr{irq261: > ix1:que } > 11 root -92 - 0K 416K CPU3 3 8:03 11.77% intr{irq259: > ix0:que } > 11 root -92 - 0K 416K CPU1 1 5:37 9.86% intr{irq257: > ix0:que } > 11 root -92 - 0K 416K WAIT 2 8:14 9.28% intr{irq258: > ix0:que } > 11 root -92 - 0K 416K RUN 0 7:07 5.18% intr{irq256: > ix0:que } > 0 root -92 0 0K 352K - 2 1:12 1.17% kernel{ix1 > que} > 12 root -16 - 0K 64K sleep 0 1:02 1.07% > ng_queue{ng_queue1} > 12 root -16 - 0K 64K sleep 0 1:02 0.88% > ng_queue{ng_queue0} > 12 root -16 - 0K 64K sleep 2 1:02 0.78% > ng_queue{ng_queue3} > 12 root -16 - 0K 64K sleep 0 1:02 0.78% > ng_queue{ng_queue2} > 0 root -92 0 0K 352K - 3 0:58 0.39% kernel{ix0 > que} > 11 root -60 - 0K 416K WAIT 3 5:21 0.00% intr{swi4: > clock} > 0 root -16 0 0K 352K sched 1 0:54 0.00% > kernel{swapper} > 14320 root 20 0 78528K 32900K select 1 0:52 0.00% mpd5{mpd5} > 0 root -92 0 0K 352K - 0 0:28 0.00% kernel{ix0 > que} > 0 root -92 0 0K 352K - 2 0:25 0.00% > kernel{dummynet} > 2382 root 20 0 18832K 4060K select 3 0:21 0.00% zebra > 0 root -92 0 0K 352K - 2 0:17 0.00% kernel{ix0 > que} > 14 root -16 - 0K 16K - 2 0:12 0.00% yarrow > 0 root -92 0 0K 352K - 3 0:07 0.00% kernel{ix0 > que} > 2388 root 20 0 25632K 6900K select 3 0:06 0.00% ospfd > 19419 www 20 0 16332K 5440K kqread 1 0:05 0.00% thttpd > 3 root -16 - 0K 16K ccb_sc 0 0:04 0.00% xpt_thrd > 9 root 16 - 0K 16K syncer 2 0:02 0.00% syncer > 0 root -92 0 0K 352K - 1 0:02 0.00% kernel{ix1 > linkq} > 3846 root 20 0 22340K 3452K select 1 0:02 0.00% ntpd > 4005 root 20 0 68024K 5976K select 1 0:01 0.00% sshd > 11 root -92 - 0K 416K WAIT 1 0:01 0.00% intr{irq262: > ix1:que } > 11 root -68 - 0K 416K WAIT 2 0:01 0.00% intr{swi2: > cambio} > 13 root -8 - 0K 48K - 3 0:01 0.00% geom{g_up} > 57732 root 20 0 32136K 2920K uwait 1 0:01 0.00% > collectd{collectd} > 11 root -88 - 0K 416K WAIT 3 0:01 0.00% intr{irq14: > ata0} > 3576 root 20 0 12192K 1716K select 2 0:01 0.00% syslogd > 81248 root 20 0 68024K 5708K select 1 0:01 0.00% sshd > 13 root -8 - 0K 48K - 0 0:01 0.00% geom{g_down} > 11 root -92 - 0K 416K WAIT 3 0:01 0.00% intr{irq264: > ix1:que } > 15 root -16 - 0K 16K sdflus 3 0:00 0.00% softdepflush > 11 root -60 - 0K 416K WAIT 2 0:00 0.00% intr{swi4: > clock} > > pciconf -lvc > --------------- > ix0@pci0:1:0:0: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82599EB 10-Gigabit SFI/SFP+ Network Connection' > class = network > subclass = ethernet > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks > cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled > cap 10[a0] = PCI-Express 2 endpoint max data 128(512) link x8(x8) > cap 03[e0] = VPD > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] = Serial 1 002590ffff3c426a > ecap 000e[150] = unknown 1 > ecap 0010[160] = unknown 1 > ix1@pci0:1:0:1: class=0x020000 card=0x061115d9 chip=0x10fb8086 rev=0x01 > hdr=0x00 > vendor = 'Intel Corporation' > device = '82599EB 10-Gigabit SFI/SFP+ Network Connection' > class = network > subclass = ethernet > cap 01[40] = powerspec 3 supports D0 D3 current D0 > cap 05[50] = MSI supports 1 message, 64 bit, vector masks > cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled > cap 10[a0] = PCI-Express 2 endpoint max data 128(512) link x8(x8) > cap 03[e0] = VPD > ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected > ecap 0003[140] = Serial 1 002590ffff3c426a > ecap 000e[150] = unknown 1 > ecap 0010[160] = unknown 1 > > The question is: > ----------------- > Why there are so much interrupts? > Why the card is using only one intr on second port (ix1)? > > I've got another server with 82598EB 10 Gigabit AT CX4 Network Connection > and the situation on it is dramatically different: > > last pid: 42234; load averages: 2.14, 1.65, 1.55 > up 119+05:58:58 13:26:01 > 98 processes: 8 running, 62 sleeping, 28 waiting > CPU 0: 0.0% user, 0.0% nice, 0.0% system, 34.0% interrupt, 66.0% idle > CPU 1: 0.0% user, 0.0% nice, 0.0% system, 43.4% interrupt, 56.6% idle > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 39.6% interrupt, 60.4% idle > CPU 3: 0.0% user, 0.0% nice, 0.0% system, 35.8% interrupt, 64.2% idle > Mem: 279M Active, 294M Inact, 300M Wired, 132K Cache, 112M Buf, 2128M Free > Swap: 4096M Total, 4096M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND > 10 root 171 ki31 0K 32K RUN 0 2159.3 70.46% {idle: cpu0} > 10 root 171 ki31 0K 32K RUN 2 2166.8 69.48% {idle: cpu2} > 10 root 171 ki31 0K 32K RUN 3 2176.3 68.90% {idle: cpu3} > 10 root 171 ki31 0K 32K RUN 1 2163.1 68.55% {idle: cpu1} > 11 root -68 - 0K 248K CPU3 3 335.0H 19.38% {irq259: > ix0:que } > 11 root -68 - 0K 248K WAIT 1 335.5H 17.97% {irq257: > ix0:que } > 11 root -68 - 0K 248K CPU2 2 335.3H 17.77% {irq258: > ix0:que } > 11 root -68 - 0K 248K CPU0 0 336.2H 17.58% {irq256: > ix0:que } > 11 root -68 - 0K 248K WAIT 3 323.4H 17.09% {irq264: > ix1:que } > 11 root -68 - 0K 248K WAIT 1 323.9H 16.55% {irq262: > ix1:que } > 11 root -68 - 0K 248K WAIT 2 325.6H 16.06% {irq263: > ix1:que } > 11 root -68 - 0K 248K WAIT 0 325.9H 15.77% {irq261: > ix1:que } > > vmstat -i > ------------- > r# vmstat -i > interrupt total rate > irq19: atapci0 1667110 0 > cpu0: timer 3386721810 328 > irq256: ix0:que 0 3395843376 329 > irq257: ix0:que 1 2642665824 256 > irq258: ix0:que 2 2838302235 275 > irq259: ix0:que 3 2176207954 211 > irq260: ix0:link 18 0 > irq261: ix1:que 0 282359321 27 > irq262: ix1:que 1 3989170496 387 > irq263: ix1:que 2 375956573 36 > irq264: ix1:que 3 3966352151 384 > irq265: ix1:link 1 0 > irq266: em0:rx 0 10283114 0 > irq269: em1:rx 0 10283114 0 > cpu3: timer 3386697130 328 > cpu1: timer 3386709028 328 > cpu2: timer 3386700908 328 > Total 33235920163 3225 > > netstat > ------------ > # netstat -I ix0 -w1 > input (ix0) output > packets errs idrops bytes packets errs bytes colls > 264572 0 0 252383894 253290 0 178845781 0 > 266323 0 0 254825280 251277 0 173769218 0 > 274462 0 0 265247306 260819 0 181113304 0 > 271142 0 0 263263325 253941 0 171694438 0 > ^C > # netstat -I ix1 -w1 > input (ix1) output > packets errs idrops bytes packets errs bytes colls > 259427 0 0 183038665 275711 0 262429749 0 > 248177 0 0 171479966 264441 0 252993752 0 > 255271 0 0 185006000 266886 0 247494148 0 > 264543 0 0 190970875 275087 0 259555066 0 > ^C > > Tuning on both systems are almost the same > As You can see, 82598EB card produce about 20 times less interrupts at > about > 10 times more pps. > > Please help me to fix the problem... > > -- > View this message in context: > http://freebsd.1045724.n5.nabble.com/Too-much-interrupts-on-ixgbe-tp4931883p4931883.html > Sent from the freebsd-net mailing list archive at Nabble.com. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbc=vM6_UMXCTOUz_mTyTVpCxp61Np68ERH8tx4KE9aT8gA>