From owner-freebsd-questions@FreeBSD.ORG Tue Aug 9 16:47:11 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 581CC1065673 for ; Tue, 9 Aug 2011 16:47:11 +0000 (UTC) (envelope-from freebsd@penx.com) Received: from Elmer.dco.penx.com (elmer.dco.penx.com [174.46.214.165]) by mx1.freebsd.org (Postfix) with ESMTP id 2E3EF8FC14 for ; Tue, 9 Aug 2011 16:47:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by Elmer.dco.penx.com (8.14.5/8.14.4) with ESMTP id p79Gl76a008553 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 9 Aug 2011 10:47:10 -0600 (MDT) (envelope-from freebsd@penx.com) Date: Tue, 9 Aug 2011 10:47:07 -0600 (MDT) From: Dennis Glatting X-X-Sender: dennisg@Elmer.dco.penx.com To: freebsd-questions@freebsd.org Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: Interrupt storms (an olde but a goode) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Aug 2011 16:47:11 -0000 I am having interrupt storms reported on an ASUS RAMPAGE III Extreme MB, specifically: iirc kernel log messages: +++ /tmp/security.WAPpO605 2011-08-09 03:01:19.000000000 -0700 +interrupt storm detected on "irq16:"; throttling interrupt source +interrupt storm detected on "irq16:"; throttling interrupt source +interrupt storm detected on "irq16:"; throttling interrupt source I have read other posts and the consensus seems to be: it's your MB. I don't understand enough of what this means to make any decision, so I am looking for pointers and other input. Is it the MB? If so, what would I look for in a replacement. This system is a ZFS host with 14 (or so) 2TB disks, some SATA SSDs, and a Revo SSD, which is kind of nice BTW. The disks are connected by the onboard controller and an Aerca board, which seems to work fairly well, BTW. Anyway, some data follows. Some data: iirc> uname -a FreeBSD iirc 8.2-STABLE FreeBSD 8.2-STABLE #28: Thu Aug 4 15:20:08 PDT 2011 root@iirc:/usr/src/sys/amd64/compile/IIRC amd64 iirc> vmstat -i interrupt total rate irq1: atkbd0 155 0 irq16: uhci0++ 377021223 920 irq18: ehci0 uhci5 5977 0 irq19: uhci2 uhci4 12850966 31 irq23: uhci3 ehci1 2 0 irq26: arcmsr0 353585806 863 cpu0: timer 812824701 1984 irq256: ahci0 207 0 irq257: em0 1382270 3 irq258: ahci2 64849716 158 cpu1: timer 812792767 1984 cpu3: timer 812792750 1984 cpu2: timer 812792693 1984 cpu6: timer 812792752 1984 cpu5: timer 812792749 1984 cpu7: timer 812792779 1984 cpu4: timer 812792637 1984 Total 7312070150 17848 iirc> grep uhci0 /var/run/dmesg.boot uhci0: port 0x9480-0x949f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup = 0x2f00 usbus1: on uhci0 iirc> grep irq /var/run/dmesg.boot |grep 16 uhci0: port 0x9480-0x949f irq 16 at device 26.0 on pci0 pcib7: irq 16 at device 0.0 on pci7 atapci0: port 0xec00-0xec0f mem 0xfbeffc00-0xfbeffc7f,0xfbef0000-0xfbef7fff irq 16 at device 0.0 on pci8 atapci1: port 0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f mem 0xfbdfe000-0xfbdfffff irq 16 at device 0.0 on pci6 iirc> cat /boot/loader.conf.local coretemp_load="YES" vboxdrv_load="YES" atapicam_load="YES" ahci_load="YES" ipmi_load="YES" xhci_load="YES" iirc> zpool status pool: disk-1 state: ONLINE scan: scrub repaired 0 in 4h19m with 0 errors on Sun Aug 7 07:22:35 2011 config: NAME STATE READ WRITE CKSUM disk-1 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 logs ada0 ONLINE 0 0 0 cache ad4 ONLINE 0 0 0 errors: No known data errors pool: disk-2 state: ONLINE scan: scrub repaired 0 in 0h4m with 0 errors on Sun Aug 7 03:07:13 2011 config: NAME STATE READ WRITE CKSUM disk-2 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 cache ad6 ONLINE 0 0 0 errors: No known data errors iirc> df -h Filesystem Size Used Avail Capacity Mounted on /dev/da0s1a 840G 15G 757G 2% / devfs 1.0k 1.0k 0B 100% /dev disk-1 2.3T 104G 2.2T 4% /disk-1 disk-2 1.8T 21G 1.8T 1% /disk-2 disk-1/word.lists 10T 8.4T 2.2T 79% /disk-1/word.lists ln:/ExternalDisk/NFS 3.6T 1.7G 3.6T 0% /NFS iirc> cat /var/run/dmesg.boot Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #28: Thu Aug 4 15:20:08 PDT 2011 root@iirc:/usr/src/sys/amd64/compile/IIRC amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM) i7 CPU 940 @ 2.93GHz (2960.51-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x106a4 Family = 6 Model = 1a Stepping = 4 Features=0xbfebfbff Features2=0x98e3bd AMD Features=0x28100800 AMD Features2=0x1 TSC: P-state invariant real memory = 12884901888 (12288 MB) avail memory = 12371128320 (11798 MB) ACPI APIC Table: <051211 APIC1500> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard kbd1 at kbdmux0 acpi0: <051211 XSDT1500> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, bff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 ahci0: port 0xac00-0xac07,0xa880-0xa883,0xa800-0xa807,0xa480-0xa483,0xa400-0xa40f mem 0xfb9ff800-0xfb9fffff irq 28 at device 0.0 on pci1 ahci0: [ITHREAD] ahci0: AHCI v1.20 with 8 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich0: [ITHREAD] ahcich1: at channel 1 on ahci0 ahcich1: [ITHREAD] ahcich2: at channel 2 on ahci0 ahcich2: [ITHREAD] ahcich3: at channel 3 on ahci0 ahcich3: [ITHREAD] ahcich4: at channel 4 on ahci0 ahcich4: [ITHREAD] ahcich5: at channel 5 on ahci0 ahcich5: [ITHREAD] ahcich6: at channel 6 on ahci0 ahcich6: [ITHREAD] ahcich7: at channel 7 on ahci0 ahcich7: [ITHREAD] pcib2: at device 2.0 on pci0 pci2: on pcib2 xhci0: mem 0xfbafe000-0xfbafffff irq 29 at device 0.0 on pci2 xhci0: [ITHREAD] xhci0: 32 byte context size. usbus0 on xhci0 pcib3: at device 3.0 on pci0 pci3: on pcib3 vgapci0: port 0xb000-0xb0ff mem 0xd0000000-0xdfffffff,0xfbbe0000-0xfbbeffff irq 24 at device 0.0 on pci3 pci3: at device 0.1 (no driver attached) pcib4: at device 5.0 on pci0 pci4: on pcib4 arcmsr0: port 0xc800-0xc8ff mem 0xfbcf0000-0xfbcfffff,0xfbc80000-0xfbcbffff irq 26 at device 0.0 on pci4 ARECA RAID ADAPTER0: Driver Version 1.20.00.21 2010-03-03 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.49 2010-12-10 arcmsr0: [ITHREAD] pcib5: at device 7.0 on pci0 pci5: on pcib5 pci0: at device 20.0 (no driver attached) pci0: at device 20.1 (no driver attached) pci0: at device 20.2 (no driver attached) pci0: at device 20.3 (no driver attached) em0: port 0x9c00-0x9c1f mem 0xfb8e0000-0xfb8fffff,0xfb8df000-0xfb8dffff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: f4:6d:04:01:c6:fc uhci0: port 0x9480-0x949f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup = 0x2f00 usbus1: on uhci0 uhci1: port 0x9800-0x981f irq 21 at device 26.1 on pci0 uhci1: [ITHREAD] uhci1: LegSup = 0x2f00 usbus2: on uhci1 uhci2: port 0x9880-0x989f irq 19 at device 26.2 on pci0 uhci2: [ITHREAD] uhci2: LegSup = 0x2f00 usbus3: on uhci2 ehci0: mem 0xfb8de000-0xfb8de3ff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus4: EHCI version 1.0 usbus4: on ehci0 pcib6: irq 17 at device 28.0 on pci0 pci7: on pcib6 pcib7: irq 16 at device 0.0 on pci7 pci8: on pcib7 atapci0: port 0xec00-0xec0f mem 0xfbeffc00-0xfbeffc7f,0xfbef0000-0xfbef7fff irq 16 at device 0.0 on pci8 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] ata4: on atapci0 ata4: [ITHREAD] ata5: on atapci0 ata5: [ITHREAD] pcib8: irq 17 at device 28.4 on pci0 pci6: on pcib8 atapci1: port 0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f mem 0xfbdfe000-0xfbdfffff irq 16 at device 0.0 on pci6 atapci1: [ITHREAD] ahci1: on atapci1 ahci1: [ITHREAD] ahci1: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported ahcich8: at channel 0 on ahci1 ahcich8: [ITHREAD] ahcich9: at channel 1 on ahci1 ahcich9: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] uhci3: port 0x9000-0x901f irq 23 at device 29.0 on pci0 uhci3: [ITHREAD] uhci3: LegSup = 0x3f00 usbus5: on uhci3 uhci4: port 0x9080-0x909f irq 19 at device 29.1 on pci0 uhci4: [ITHREAD] uhci4: LegSup = 0x2f00 usbus6: on uhci4 uhci5: port 0x9400-0x941f irq 18 at device 29.2 on pci0 uhci5: [ITHREAD] uhci5: LegSup = 0x2f00 usbus7: on uhci5 ehci1: mem 0xfb8dd000-0xfb8dd3ff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus8: waiting for BIOS to give up control usbus8: timed out waiting for BIOS usbus8: EHCI version 1.0 usbus8: on ehci1 pcib9: at device 30.0 on pci0 pci9: on pcib9 isab0: at device 31.0 on pci0 isa0: on isab0 ahci2: port 0x8880-0x8887,0x8800-0x8803,0x8480-0x8487,0x8400-0x8403,0x8080-0x809f mem 0xfb8db000-0xfb8db7ff irq 20 at device 31.2 on pci0 ahci2: [ITHREAD] ahci2: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported ahcich10: at channel 0 on ahci2 ahcich10: [ITHREAD] ahcich11: at channel 1 on ahci2 ahcich11: [ITHREAD] ahcich12: at channel 2 on ahci2 ahcich12: [ITHREAD] ahcich13: at channel 3 on ahci2 ahcich13: [ITHREAD] ahcich14: at channel 4 on ahci2 ahcich14: [ITHREAD] ahcich15: at channel 5 on ahci2 ahcich15: [ITHREAD] pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] qpi0: on motherboard pcib10: pcibus 255 on qpi0 pci255: on pcib10 orm0: at iomem 0xd0000-0xd2fff,0xd4800-0xd77ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range coretemp0: on cpu0 est0: on cpu0 p4tcc0: on cpu0 coretemp1: on cpu1 est1: on cpu1 p4tcc1: on cpu1 coretemp2: on cpu2 est2: on cpu2 p4tcc2: on cpu2 coretemp3: on cpu3 est3: on cpu3 p4tcc3: on cpu3 coretemp4: on cpu4 est4: on cpu4 p4tcc4: on cpu4 coretemp5: on cpu5 est5: on cpu5 p4tcc5: on cpu5 coretemp6: on cpu6 est6: on cpu6 p4tcc6: on cpu6 coretemp7: on cpu7 est7: on cpu7 p4tcc7: on cpu7 Timecounters tick every 1.000 msec vboxdrv: fAsync=0 offMin=0x334 offMax=0x574 usbus0: 5.0Gbps Super Speed USB v3.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 480Mbps High Speed USB v2.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 12Mbps Full Speed USB v1.0 usbus8: 480Mbps High Speed USB v2.0 ugen0.1: <0x1033> at usbus0 uhub0: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 ugen8.1: at usbus8 uhub8: on usbus8 uhub0: 4 ports with 4 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered uhub7: 2 ports with 2 removable, self powered uhub4: 6 ports with 6 removable, self powered uhub8: 6 ports with 6 removable, self powered ugen3.2: at usbus3 ugen7.2: at usbus7 ums0: on usbus7 ums0: 3 buttons and [XYZ] coordinates ID=0 ugen6.2: at usbus6 ahcich7: Poll timeout on slot 0 port 0 ahcich7: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd 50 serr 00000000 ad4: 57241MB at ata2-master UDMA100 SATA 3Gb/s ad6: 57241MB at ata3-master UDMA100 SATA 3Gb/s da0 at arcmsr0 bus 0 scbus8 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da0: Command Queueing enabled da0: 953869MB (1953523712 512 byte sectors: 255H 63S/T 121601C) da1 at arcmsr0 bus 0 scbus8 target 0 lun 1 da1: Fixed Direct Access SCSI-5 device da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da1: Command Queueing enabled da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da2 at arcmsr0 bus 0 scbus8 target 0 lun 2 da2: Fixed Direct Access SCSI-5 device da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da2: Command Queueing enabled da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da3 at arcmsr0 bus 0 scbus8 target 0 lun 3 da3: Fixed Direct Access SCSI-5 device da3: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da3: Command Queueing enabled da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da4 at arcmsr0 bus 0 scbus8 target 0 lun 4 da4: Fixed Direct Access SCSI-5 device da4: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da4: Command Queueing enabled da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da5 at arcmsr0 bus 0 scbus8 target 0 lun 5 da5: Fixed Direct Access SCSI-5 device da5: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da5: Command Queueing enabled da5: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da6 at arcmsr0 bus 0 scbus8 target 0 lun 6 da6: Fixed Direct Access SCSI-5 device da6: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) da6: Command Queueing enabled da6: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 57241MB (117231408 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich10 bus 0 scbus11 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2 at ahcich11 bus 0 scbus12 target 0 lun 0 ada2: ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3 at ahcich12 bus 0 scbus13 target 0 lun 0 ada3: ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada4 at ahcich13 bus 0 scbus14 target 0 lun 0 ada4: ATA-8 SATA 2.x device ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada5 at ahcich14 bus 0 scbus15 target 0 lun 0 ada5: ATA-8 SATA 2.x device ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada6 at ahcich15 bus 0 scbus16 target 0 lun 0 ada6: ATA-8 SATA 3.x device ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada6: Command Queueing enabled ada6: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) pass8 at arcmsr0 bus 0 scbus8 target 16 lun 0 pass8: Fixed Processor SCSI-0 device SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! cd0 at ahcich9 bus 0 scSbMuPs: 1A0 Pt CaPrUg #et 60 lLuna u0nched! cd0: Removable CD-ROM SCSI-0 device cd0: 150.000MB/s transfers (SATA 1.x, UDMA6, ATAPI 12bytes, PIO 8192bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed SMP: AP CPU #5 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #4 Launched! Trying to mount root from ufs:/dev/da0s1a ZFS filesystem version 5 ZFS storage pool version 28 WARNING: reducing size to maximum of 67108864 blocks per swap unit vboxnet0: Ethernet address: 0a:00:27:00:00:00 ubt0: on usbus6 WARNING: attempt to domain_add(bluetooth) after domainfinalize() WARNING: attempt to domain_add(netgraph) after domainfinalize() em0: link state changed to UP