From owner-freebsd-fs@FreeBSD.ORG Tue Sep 9 18:29:12 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B97C3FEA for ; Tue, 9 Sep 2014 18:29:12 +0000 (UTC) Received: from mail-qg0-x232.google.com (mail-qg0-x232.google.com [IPv6:2607:f8b0:400d:c04::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 56D97234 for ; Tue, 9 Sep 2014 18:29:12 +0000 (UTC) Received: by mail-qg0-f50.google.com with SMTP id z60so2403831qgd.23 for ; Tue, 09 Sep 2014 11:29:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bayphoto.com; s=google; h=mime-version:from:date:message-id:subject:to:content-type; bh=z0nsq3cQE4Do8LOaQpIpRIZ/gaE2T5sy2LDFSLhPM/U=; b=Meus0DoCybB1EjojJah3mZqruX1JENJ+TIhozERZK4Oxh70y/z0LrCnFtiZEQKDXTU OpUXi0aD1uMbdcIJPK7T2o+bREeQ67hznS8Mll18Mr9SEWkjAaoPacqmCaKa47t3rYx+ Ql7oKUBgql9qqxyzcORmV+BJ3AMrgcLylL5H8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=z0nsq3cQE4Do8LOaQpIpRIZ/gaE2T5sy2LDFSLhPM/U=; b=TddVCocApv+xY2SDswHsF8Czqs45zaAT3qGtdsNpcJgOBh3JxYqFBx/e7kK0ipH3aB UMNLzmrSAb3sHShzPR+cgOjQaP73nIPYROcZIRw7cKq8sGX+YK/Wk5FA8KHrZQOq+e4C cQv1FmOHbujG55XBUsTtH56OkGXhOKx2li5ywYiDGGmL3Fsabq/nD8tFHyW2pzuZRqbL q0lQbOHdLP5gwk7JD2GemcBj8TXOeH+zkaKQecIKstGyvvBDc0LSjNlUrIuQGaSEg6qY WEkD+nnmj3b8U/Ykt/usHDym/IaydKqxNuy/79JG8eb7/qmzSLm3TYn5abbrPOqJBFrg 9+EA== X-Gm-Message-State: ALoCoQmxhlYA3zR3/Mf71Z3vzjoRHANzZeQGVear0tfn40OqiTj9lViGJM4fole7fB2lrFuLKCZn X-Received: by 10.140.31.75 with SMTP id e69mr51935841qge.2.1410287351009; Tue, 09 Sep 2014 11:29:11 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.86.230 with HTTP; Tue, 9 Sep 2014 11:28:50 -0700 (PDT) From: Mike Carlson Date: Tue, 9 Sep 2014 11:28:50 -0700 Message-ID: Subject: ZFS Panic (3rd time) To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 18:29:12 -0000 Issue: Server panic'd around 7:44PST last night, cause is unknown (no auto dumpdev, zfs root) The server has two zfs pools, a mirrored zroot, and a RAIDZ2 data volume (with about a dozen zfs filesystems in use) At this point, simply running: zpool status data results in a panic. I can't run a zpool history either to display the config, but it was a RAIDz2 with a spare Background: This is the 3rd rebuild of a large 20TB file server. We first had issues when we upgraded from 9.1 -> 10.0 (we were hoping to resolve some zfs related deadlocks) I've rebuilt this server for the 3rd time (the last time, the LSI controller was updated to the latest firmware) and after around two months of uptime, the system unexpectedly panic'd A lot of the data was getting sync'd over to a backup server, but one volume was not. I would greatly appreciate some help if I can possibly get one of the volumes back. zdb commands don't work on the pool,: ~ # zdb -uuuC data zdb: can't open 'data': No such file or directory Again, I thank anyone in advanced for taking the time to read this disaster again Panic: GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-RELEASE-p7 #0: Tue Sep 9 09:06:18 PDT 2014 root@zfs-2.discdrive.bayphoto.com:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 CPU: AMD Opteron(tm) Processor 4386 (3100.07-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x600f20 Family = 0x15 Model = 0x2 Stepping = 0 Features=0x178bfbff Features2=0x3e98320b AMD Features=0x2e500800 AMD Features2=0x1ebbfff Standard Extended Features=0x8 TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 66707759104 (63617 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <103012 APIC1341> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 8 core(s) cpu0 (BSP): APIC ID: 16 cpu1 (AP): APIC ID: 17 cpu2 (AP): APIC ID: 18 cpu3 (AP): APIC ID: 19 cpu4 (AP): APIC ID: 20 cpu5 (AP): APIC ID: 21 cpu6 (AP): APIC ID: 22 cpu7 (AP): APIC ID: 23 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-55 on motherboard random: initialized kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of fec00000, 1000 (3) failed acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of fec10000, 20 (3) failed acpi0: reservation of ffe50000, e060 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dff00000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 52 at device 3.0 on pci0 pci1: on pcib1 mps0: port 0xc000-0xc0ff mem 0xfe83c000-0xfe83ffff,0xfe840000-0xfe87ffff irq 28 at device 0.0 on pci1 mps0: Firmware: 19.00.00.00, Driver: 16.00.00.00-fbsd mps0: IOCCapabilities: 1285c pcib2: irq 53 at device 9.0 on pci0 pci2: on pcib2 em0: port 0xd800-0xd81f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 48 at device 0.0 on pci2 em0: Using MSIX interrupts with 3 vectors em0: Ethernet address: 00:25:90:a1:04:ea pcib3: irq 54 at device 10.0 on pci0 pci3: on pcib3 em1: port 0xe800-0xe81f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 47 at device 0.0 on pci3 em1: Using MSIX interrupts with 3 vectors em1: Ethernet address: 00:25:90:a1:04:eb ahci0: port 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f mem 0xfebfbc00-0xfebfbfff irq 22 at device 17.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ohci0: mem 0xfebf9000-0xfebf9fff irq 16 at device 18.0 on pci0 usbus0 on ohci0 ohci1: mem 0xfebfa000-0xfebfafff irq 16 at device 18.1 on pci0 usbus1 on ohci1 ehci0: mem 0xfebfb800-0xfebfb8ff irq 17 at device 18.2 on pci0 usbus2: EHCI version 1.0 usbus2 on ehci0 ohci2: mem 0xfebf7000-0xfebf7fff irq 18 at device 19.0 on pci0 usbus3 on ohci2 ohci3: mem 0xfebf8000-0xfebf8fff irq 18 at device 19.1 on pci0 usbus4 on ohci3 ehci1: mem 0xfebfb400-0xfebfb4ff irq 19 at device 19.2 on pci0 usbus5: EHCI version 1.0 usbus5 on ehci1 pci0: at device 20.0 (no driver attached) atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib4: at device 20.4 on pci0 pci4: on pcib4 vgapci0: mem 0xfc000000-0xfcffffff,0xfdffc000-0xfdffffff,0xfe000000-0xfe7fffff irq 20 at device 4.0 on pci4 vgapci0: Boot video device ohci4: mem 0xfebf6000-0xfebf6fff irq 18 at device 20.5 on pci0 usbus6 on ohci4 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 orm0: at iomem 0xc0000-0xc7fff,0xcd800-0xce7ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range acpi_throttle0: on cpu0 hwpstate0: on cpu0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec random: unblocking device. usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 usbus6: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen4.1: at usbus4 uhub3: on usbus4 ugen3.1: at usbus3 uhub4: on usbus3 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 uhub6: 2 ports with 2 removable, self powered uhub0: 3 ports with 3 removable, self powered uhub1: 3 ports with 3 removable, self powered uhub3: 3 ports with 3 removable, self powered uhub4: 3 ports with 3 removable, self powered uhub2: 6 ports with 6 removable, self powered uhub5: 6 ports with 6 removable, self powered ugen0.2: at usbus0 ukbd0: on usbus0 kbd2 at ukbd0 ugen2.2: at usbus2 umass0: on usbus2 umass0: SCSI over Bulk-Only; quirks = 0x4000 umass0:7:0:-1: Attached to scbus7 ses0 at mps0 bus 0 scbus0 target 32 lun 0 ses0: Fixed Enclosure Services SCSI-5 device ses0: Serial Number ses0: 600.000MB/s transfers ses0: Command Queueing enabled ses0: SCSI-3 ENC Device ses1 at mps0 bus 0 scbus0 target 39 lun 0 ses1: Fixed Enclosure Services SCSI-5 device ses1: Serial Number ses1: 600.000MB/s transfers ses1: Command Queueing enabled ses1: SCSI-3 ENC Device da1 at mps0 bus 0 scbus0 target 11 lun 0 da1: Fixed Direct Access SCSI-6 device da1: Serial Number PK2131P6G2Y3GP da1: 600.000MB/s transfers da1: Command Queueing enabled da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da2 at mps0 bus 0 scbus0 target 12 lun 0 da2: Fixed Direct Access SCSI-6 device da2: Serial Number PK2131P6G2XXPP da2: 600.000MB/s transfers da2: Command Queueing enabled da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da3 at mps0 bus 0 scbus0 target 13 lun 0 da3: Fixed Direct Access SCSI-6 device da3: Serial Number PK2131P6G2ZDVP da3: 600.000MB/s transfers da3: Command Queueing enabled da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da4 at mps0 bus 0 scbus0 target 14 lun 0 da4: Fixed Direct Access SCSI-6 device da4: Serial Number PK2131P6G2XWHP da4: 600.000MB/s transfers da4: Command Queueing enabled da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da5 at mps0 bus 0 scbus0 target 15 lun 0 da5: Fixed Direct Access SCSI-6 device da5: Serial Number PK2131P6G2XXHP da5: 600.000MB/s transfers da5: Command Queueing enabled da5: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da6 at mps0 bus 0 scbus0 target 16 lun 0 da6: Fixed Direct Access SCSI-6 device da6: Serial Number PK2131P6G2VBLP da6: 600.000MB/s transfers da6: Command Queueing enabled da6: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da7 at mps0 bus 0 scbus0 target 17 lun 0 da7: Fixed Direct Access SCSI-6 device da7: Serial Number PK2131P6G2XVWP da7: 600.000MB/s transfers da7: Command Queueing enabled da7: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da8 at mps0 bus 0 scbus0 target 18 lun 0 da8: Fixed Direct Access SCSI-6 device da8: Serial Number PK2131P6G2L66P da8: 600.000MB/s transfers da8: Command Queueing enabled da8: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da9 at mps0 bus 0 scbus0 target 19 lun 0 da9: Fixed Direct Access SCSI-6 device da9: Serial Number PK2131P6G2XY8P da9: 600.000MB/s transfers da9: Command Queueing enabled da9: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da10 at mps0 bus 0 scbus0 target 20 lun 0 da10: Fixed Direct Access SCSI-6 device da10: Serial Number PK2131P6G1YGBP da10: 600.000MB/s transfers da10: Command Queueing enabled da10: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da11 at mps0 bus 0 scbus0 target 21 lun 0 da11: Fixed Direct Access SCSI-6 device da11: Serial Number PK2131P6G2XXRP da11: 600.000MB/s transfers da11: Command Queueing enabled da11: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da0 at mps0 bus 0 scbus0 target 10 lun 0 da0: Fixed Direct Access SCSI-6 device da0: Serial Number A165A011311000011 da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 228936MB (468862128 512 byte sectors: 255H 63S/T 29185C) da12 at mps0 bus 0 scbus0 target 22 lun 0 da12: Fixed Direct Access SCSI-6 device da12: Serial Number PK2131P6G2Y92P da12: 600.000MB/s transfers da12: Command Queueing enabled da12: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da13 at mps0 bus 0 scbus0 target 23 lun 0 da13: Fixed Direct Access SCSI-6 device da13: Serial Number PK2131P6G2XXXP da13: 600.000MB/s transfers da13: Command Queueing enabled da13: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da14 at mps0 bus 0 scbus0 target 24 lun 0 da14: Fixed Direct Access SCSI-6 device da14: Serial Number PK2131P6G2Y3NP da14: 600.000MB/s transfers da14: Command Queueing enabled da14: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da15 at mps0 bus 0 scbus0 target 25 lun 0 da15: Fixed Direct Access SCSI-6 device da15: Serial Number PK2131P6G2VBXP da15: 600.000MB/s transfers da15: Command Queueing enabled da15: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da16 at mps0 bus 0 scbus0 target 26 lun 0 da16: Fixed Direct Access SCSI-6 device da16: Serial Number PK2131P6G2X75P da16: 600.000MB/s transfers da16: Command Queueing enabled da16: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da17 at mps0 bus 0 scbus0 target 27 lun 0 da17: Fixed Direct Access SCSI-6 device da17: Serial Number PK2131P6G2VD6P da17: 600.000MB/s transfers da17: Command Queueing enabled da17: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da18 at mps0 bus 0 scbus0 target 28 lun 0 da18: Fixed Direct Access SCSI-6 device da18: Serial Number PK2131P6G2VBWP da18: 600.000MB/s transfers da18: Command Queueing enabled da18: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da19 at mps0 bus 0 scbus0 target 29 lun 0 da19: Fixed Direct Access SCSI-6 device da19: Serial Number PK2131P6G2VDHP da19: 600.000MB/s transfers da19: Command Queueing enabled da19: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da20 at mps0 bus 0 scbus0 target 30 lun 0 da20: Fixed Direct Access SCSI-6 device da20: Serial Number PK2131P6G2VAPP da20: 600.000MB/s transfers da20: Command Queueing enabled da20: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da21 at mps0 bus 0 scbus0 target 31 lun 0 da21: Fixed Direct Access SCSI-6 device da21: Serial Number PK2131P6G2XWSP da21: 600.000MB/s transfers da21: Command Queueing enabled da21: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da22 at mps0 bus 0 scbus0 target 33 lun 0 da22: Fixed Direct Access SCSI-6 device da22: Serial Number PK2131P6G2ZDRP da22: 600.000MB/s transfers da22: Command Queueing enabled da22: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da28 at mps0 bus 0 scbus0 target 40 lun 0 da28: Fixed Direct Access SCSI-6 device da28: Serial Number 000000001247091DCBF3 da28: 600.000MB/s transfers da28: Command Queueing enabled da28: 244198MB (500118192 512 byte sectors: 255H 63S/T 31130C) da28: quirks=0x8<4K> da29 at mps0 bus 0 scbus0 target 41 lun 0 da29: Fixed Direct Access SCSI-6 device da29: Serial Number 00000000130109242782 da29: 600.000MB/s transfers da29: Command Queueing enabled da29: 244198MB (500118192 512 byte sectors: 255H 63S/T 31130C) da29: quirks=0x8<4K> da27 at mps0 bus 0 scbus0 target 38 lun 0 da27: Fixed Direct Access SCSI-6 device da27: Serial Number WD-WCAT1F522653 da27: 300.000MB/s transfers da27: Command Queueing enabled da27: 238418MB (488281250 512 byte sectors: 255H 63S/T 30394C) da23 at mps0 bus 0 scbus0 target 34 lun 0 da23: Fixed Direct Access SCSI-6 device da23: Serial Number PK2131P6G2VBPP da23: 600.000MB/s transfers da23: Command Queueing enabled da23: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da24 at mps0 bus 0 scbus0 target 35 lun 0 da24: Fixed Direct Access SCSI-6 device da24: Serial Number PK2131P6G2Y37P da24: 600.000MB/s transfers da24: Command Queueing enabled da24: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da25 at mps0 bus 0 scbus0 target 36 lun 0 da25: Fixed Direct Access SCSI-6 device da25: Serial Number PK2131P6G2Y90P da25: 600.000MB/s transfers da25: Command Queueing enabled da25: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da26 at mps0 bus 0 scbus0 target 37 lun 0 da26: Fixed Direct Access SCSI-6 device da26: Serial Number PK2131P6G2Y26P da26: 600.000MB/s transfers da26: Command Queueing enabled da26: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da30 at umass-sim0 bus 0 scbus7 target 0 lun 0 da30: Fixed Direct Access SCSI-4 device da30: Serial Number 0010101E4003DDD6 da30: 40.000MB/s transfers da30: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C) da30: quirks=0x2 ses1: da22: Element descriptor: 'Slot 01' ses1: da22: SAS Device Slot Element: 1 Phys at Slot 0 ses1: phy 0: SATA device ses1: phy 0: parent 50030480003d1ebf addr 50030480003d1eac ses1: da23: Element descriptor: 'Slot 02' ses1: da24: Element descriptor: 'Slot 03' ses1: da24: SAS Device Slot Element: 1 Phys at Slot 2 ses1: phy 0: SATA device ses1: phy 0: parent 50030480003d1ebf addr 50030480003d1eae ses1: da25: Element descriptor: 'Slot 04' ses1: da25: SAS Device Slot Element: 1 Phys at Slot 3 ses1: phy 0: SATA device ses1: phy 0: parent 50030480003d1ebf addr 50030480003d1eaf ses1: da26: Element descriptor: 'Slot 05' ses1: da26: SAS Device Slot Element: 1 Phys at Slot 4 ses1: phy 0: SATA device ses1: phy 0: parent 50030480003d1ebf addr 50030480003d1eb0 ses1: da27: Element descriptor: 'Slot 12' ses1: da27: SAS Device Slot Element: 1 Phys at Slot 11 ses1: phy 0: SATA device ses1: phy 0: parent 50030480003d1ebf addr 50030480003d1eb7 ses0: da28: Element descriptor: 'Slot 01' ses0: da28: SAS Device Slot Element: 1 Phys at Slot 0 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c0c ses0: da29: Element descriptor: 'Slot 02' ses0: da29: SAS Device Slot Element: 1 Phys at Slot 1 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c0d ses0: da0,pass0: Element descriptor: 'Slot 03' ses0: da0,pass0: SAS Device Slot Element: 1 Phys at Slot 2 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c0e ses0: da1: Element descriptor: 'Slot 04' ses0: da1: SAS Device Slot Element: 1 Phys at Slot 3 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c0f ses0: da2: Element descriptor: 'Slot 05' ses0: da2: SAS Device Slot Element: 1 Phys at Slot 4 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c10 ses0: da3: Element descriptor: 'Slot 06' ses0: da3: SAS Device Slot Element: 1 Phys at Slot 5 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c11 ses0: da4: Element descriptor: 'Slot 07' ses0: da4: SAS Device Slot Element: 1 Phys at Slot 6 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c12 ses0: da5: Element descriptor: 'Slot 08' ses0: da5: SAS Device Slot Element: 1 Phys at Slot 7 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c13 ugen4.2: at usbus4 ses0: da6: Element descriptor: 'Slot 09' ses0: da6: SAS Device Slot Element: 1 Phys at Slot 8 ses0: phy 0: SATA device ukbd1: on usbus4 ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c14 ses0: da7: Element descriptor: 'Slot 10' ses0: da7: SAS Device Slot Element: 1 Phys at Slot 9 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c15 kbd3 at ukbd1 ses0: da8: Element descriptor: 'Slot 11' ses0: da8: SAS Device Slot Element: 1 Phys at Slot 10 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c16 ses0: da9: Element descriptor: 'Slot 12' ses0: da9: SAS Device Slot Element: 1 Phys at Slot 11 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c17 ses0: da10: Element descriptor: 'Slot 13' ses0: da10: SAS Device Slot Element: 1 Phys at Slot 12 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c18 ses0: da11: Element descriptor: 'Slot 14' ses0: da11: SAS Device Slot Element: 1 Phys at Slot 13 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c19 ses0: da12: Element descriptor: 'Slot 15' ses0: da12: SAS Device Slot Element: 1 Phys at Slot 14 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c1a ses0: da13: Element descriptor: 'Slot 16' ses0: da13: SAS Device Slot Element: 1 Phys at Slot 15 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c1b ses0: da14: Element descriptor: 'Slot 17' ses0: da14: SAS Device Slot Element: 1 Phys at Slot 16 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c1c ses0: da15: Element descriptor: 'Slot 18' ses0: da15: SAS Device Slot Element: 1 Phys at Slot 17 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c1d ses0: da16: Element descriptor: 'Slot 19' ses0: da16: SAS Device Slot Element: 1 Phys at Slot 18 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c1e ses0: da17: Element descriptor: 'Slot 20' ses0: da17: SAS Device Slot Element: 1 Phys at Slot 19 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c1f ses0: da18: Element descriptor: 'Slot 21' ses0: da18: SAS Device Slot Element: 1 Phys at Slot 20 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c20 ses0: da19: Element descriptor: 'Slot 22' ses0: da19: SAS Device Slot Element: 1 Phys at Slot 21 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c21 ses0: da20: Element descriptor: 'Slot 23' ses0: da20: SAS Device Slot Element: 1 Phys at Slot 22 ses0: phy 0: SATA device ses0: phy 0: parent 50030480003e9c3f addr 50030480003e9c22 ses0: da21: Element descriptor: 'Slot 24' Netvsc initializing... SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #4 Launched! Timecounter "TSC-low" frequency 1550032870 Hz quality 1000 Trying to mount root from zfs:zroot []... Fatal trap 12: page fault while in kernel mode Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 10 cpuid = 3; apic id = 13 fault virtual address = 0x50 fault virtual address = 0xa0 fault code = supervisor read data, page not present fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff81860336 instruction pointer = 0x20:0xffffffff8187d3a0 stack pointer = 0x28:0xfffffe104c762ab0 stack pointer = 0x28:0xfffffe104cd2e2a0 frame pointer = 0x28:0xfffffe104c762ac0 frame pointer = 0x28:0xfffffe104cd2e330 code segment = base 0x0, limit 0xfffff, type 0x1b code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = processor eflags = interrupt enabled, interrupt enabled, resume, IOPL = 0 resume, IOPL = 0 current process = 0 (system_taskq_6) current process = 670 (zpool) trap number = 12 trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xffffffff808e7e90 at kdb_backtrace+0x60 #1 0xffffffff808af975 at panic+0x155 #2 0xffffffff80c8e832 at trap_fatal+0x3a2 #3 0xffffffff80c8eb09 at trap_pfault+0x2c9 #4 0xffffffff80c8e296 at trap+0x5e6 #5 0xffffffff80c75532 at calltrap+0x8 #6 0xffffffff81866800 at vdev_mirror_child_select+0x70 #7 0xffffffff81866344 at vdev_mirror_io_start+0x234 #8 0xffffffff818803c4 at zio_vdev_io_start+0x184 #9 0xffffffff8187d796 at zio_execute+0x136 #10 0xffffffff8180ddec at arc_read+0x87c #11 0xffffffff818268f1 at traverse_prefetcher+0x141 #12 0xffffffff81825d14 at traverse_visitbp+0x1c4 #13 0xffffffff8182677f at traverse_dnode+0xef #14 0xffffffff81826487 at traverse_visitbp+0x937 #15 0xffffffff81825ee3 at traverse_visitbp+0x393 #16 0xffffffff81825ee3 at traverse_visitbp+0x393 #17 0xffffffff81825ee3 at traverse_visitbp+0x393 Uptime: 25m1s Dumping 2296 out of 65493 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols #0 doadump (textdump=) at pcpu.h:219 219 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) Backtrace: (kgdb) bt #0 doadump (textdump=) at pcpu.h:219 #1 0xffffffff808af5f0 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xffffffff808af9b4 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xffffffff80c8e832 in trap_fatal (frame=, eva=) at /usr/src/sys/amd64/amd64/trap.c:882 #4 0xffffffff80c8eb09 in trap_pfault (frame=0xfffffe104c762a00, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699 #5 0xffffffff80c8e296 in trap (frame=0xfffffe104c762a00) at /usr/src/sys/amd64/amd64/trap.c:463 #6 0xffffffff80c75532 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #7 0xffffffff81860336 in vdev_rele (vd=0x0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:1556 #8 0xffffffff81866800 in vdev_mirror_scrub_done (zio=0x3) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:193 #9 0xffffffff81866344 in vdev_mirror_io_start (zio=0xfffff80142733d00) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:286 #10 0xffffffff818803c4 in zio_vdev_io_start (zio=0xfffff8013eb20b10) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2571 #11 0xffffffff8187d796 in zio_suspend (spa=0xfffff8000e122000, zio=0xfffff8013eb20b10) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1456 #12 0xffffffff8180ddec in arc_read (pio=0x0, spa=0xfffff8001e37d000, bp=, done=0x2, private=0x0, priority=6, zio_flags=0, arc_flags=, zb=0xfffff8001ed06558) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3371 #13 0xffffffff818268f1 in traverse_prefetcher (spa=0xfffff8001e37d000, zilog=0xf01ff, bp=, zb=, dnp=0xfffff80142733d00, arg=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:451 #14 0xffffffff81825d14 in traverse_visitbp (td=0xfffffe104c763900, dnp=0xfffffe000fe64800, bp=0xfffffe000fe64980, zb=0xfffffe104c762e88) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:250 #15 0xffffffff8182677f in traverse_dnode (td=0xfffffe104c763900, dnp=0xfffffe000fe64800, objset=110, object=26823324) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:417 #16 0xffffffff81826487 in traverse_visitbp (td=0xfffffe104c763900, dnp=0xfffffe000fe61000, bp=0xfffffe001285ea00, zb=0xfffffe104c7630a8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:309 #17 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900, dnp=0xfffff8013ed8f000, bp=0xfffffe0012867a00, zb=0xfffffe104c7631d8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #18 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900, dnp=0xfffff8013ed8f000, bp=0xfffffe0012842980, zb=0xfffffe104c763308) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #19 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900, dnp=0xfffff8013ed8f000, bp=0xfffffe0012848000, zb=0xfffffe104c763438) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #20 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900, dnp=0xfffff8013ed8f000, bp=0xfffffe000fe6d000, zb=0xfffffe104c763568) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #21 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900, dnp=0xfffff8013ed8f000, bp=0xfffffe001282d000, zb=0xfffffe104c763698) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #22 0xffffffff81825ee3 in traverse_visitbp (td=0xfffffe104c763900, dnp=0xfffff8013ed8f000, bp=0xfffff8013ed8f040, zb=0xfffffe104c763758) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #23 0xffffffff81826714 in traverse_dnode (td=0xfffffe104c763900, dnp=0xfffff8013ed8f000, objset=110, object=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:407 #24 0xffffffff81826190 in traverse_visitbp (td=0xfffffe104c763900, dnp=0x0, bp=0xfffff8013ed7ea80, zb=0xfffffe104c7638e0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:338 #25 0xffffffff81825af6 in traverse_prefetch_thread (arg=0xfffffe104cd2f0e0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:470 #26 0xffffffff817fcc00 in taskq_run (arg=0xfffff801445bef30, pending=983551) at /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c:109 #27 0xffffffff808f5c26 in taskqueue_run_locked (queue=0xfffff8000e0eba00) at /usr/src/sys/kern/subr_taskqueue.c:333 #28 0xffffffff808f64a8 in taskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_taskqueue.c:535 #29 0xffffffff80881a4a in fork_exit (callout=0xffffffff808f6400 , arg=0xfffff8000e10aac0, frame=0xfffffe104c763a40) at /usr/src/sys/kern/kern_fork.c:995 #30 0xffffffff80c75a6e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #31 0x0000000000000000 in ?? () Current language: auto; currently minimal