Date: Wed, 31 Mar 2010 12:20:31 GMT From: "Bartek K." <darklca@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: amd64/145238: kernel panic on zpool clear tank Message-ID: <201003311220.o2VCKVG6092630@www.freebsd.org> Resent-Message-ID: <201003311230.o2VCU8eH083754@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 145238
>Category: amd64
>Synopsis: kernel panic on zpool clear tank
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-amd64
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Wed Mar 31 12:30:07 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator: Bartek K.
>Release: 8.0
>Organization:
>Environment:
FreeBSD backuptest 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:02:08 UTC 2009 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
3ware9650 24 port M8 controller ( all drives configured as single drives )
after power failure, i can`t get online backup pool:
pool: backup
state: FAULTED
status: An intent log record could not be read.
Waiting for adminstrator intervention to fix the faulted pool.
action: Either restore the affected device(s) and run 'zpool online',
or ignore the intent log records by running 'zpool clear'.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
backup FAULTED 0 0 1 bad intent log
raidz1 ONLINE 0 0 2
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
da5 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da6 ONLINE 0 0 0
da7 ONLINE 0 0 0
da8 ONLINE 0 0 0
da9 ONLINE 0 0 0
da10 ONLINE 0 0 0
raidz1 ONLINE 0 0 2
da11 ONLINE 0 0 0
da12 ONLINE 0 0 0
da13 ONLINE 0 0 0
da14 ONLINE 0 0 0
da15 ONLINE 0 0 0
raidz1 ONLINE 0 0 2
da16 ONLINE 0 0 0
da17 ONLINE 0 0 0
da18 ONLINE 0 0 0
da19 ONLINE 0 0 0
da20 ONLINE 0 0 0
if i try to execute "zpool clear backup", its instantly panic:
panic:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x8000000000
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80198526
stack pointer = 0x28:0xffffff820aab9e80
frame pointer = 0x28:0xffffff820aab9f90
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 1436 (txg_thread_enter)
trap number = 12
panic: page fault
cpuid = 1
Uptime: 3m31s
dmesg:
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RELEASE-p2 #1: Wed Mar 31 12:03:06 UTC 2010
root@backuptest:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (2000.01-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x1067a Stepping = 10
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x40ce33d<SSE3,DTES64,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 19327352832 (18432 MB)
avail memory = 18629591040 (17766 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: <SMCI SMCISLP2> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2000-0x201f mem 0xda020000-0xda03ffff,0xda000000-0xda01ffff irq 18 at device 0.0 on pci6
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:30:48:62:35:9e
em1: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x2020-0x203f mem 0xda060000-0xda07ffff,0xda040000-0xda05ffff irq 19 at device 0.1 on pci6
em1: Using MSI interrupt
em1: [FILTER]
em1: Ethernet address: 00:30:48:62:35:9f
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
3ware device driver for 9000 series storage controllers, version: 3.70.05.001
twa0: <3ware 9000 series Storage Controller> port 0x3000-0x30ff mem 0xd8000000-0xd9ffffff,0xda300000-0xda300fff irq 16 at device 0.0 on pci8
twa0: [ITHREAD]
twa0: INFO: (0x04: 0x0001): Controller reset occurred: resets=2
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-24M8, 24 ports, Firmware FE9X 4.10.00.007, BIOS BE9X 4.08.00.002
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
em2: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x4000-0x401f mem 0xda220000-0xda23ffff,0xda200000-0xda21ffff irq 18 at device 0.0 on pci9
em2: Using MSI interrupt
em2: [FILTER]
em2: Ethernet address: 00:15:17:8a:bc:c9
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci10: <ACPI PCI bus> on pcib10
em3: <Intel(R) PRO/1000 Network Connection 6.9.14> port 0x5000-0x501f mem 0xda420000-0xda43ffff,0xda400000-0xda41ffff irq 16 at device 0.0 on pci10
em3: Using MSI interrupt
em3: [FILTER]
em3: Ethernet address: 00:15:17:8a:bb:b3
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x003b
usbus0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x0010
usbus1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x0010
usbus2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xda800000-0xda8003ff irq 17 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 63XXESB USB 2.0 controller> on ehci0
pcib11: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci11: <ACPI PCI bus> on pcib11
vgapci0: <VGA-compatible display> port 0x6000-0x60ff mem 0xd0000000-0xd7ffffff,0xda500000-0xda50ffff irq 18 at device 1.0 on pci11
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
cpu4: <ACPI CPU> on acpi0
p4tcc4: <CPU Frequency Thermal Control> on cpu4
cpu5: <ACPI CPU> on acpi0
p4tcc5: <CPU Frequency Thermal Control> on cpu5
cpu6: <ACPI CPU> on acpi0
p4tcc6: <CPU Frequency Thermal Control> on cpu6
cpu7: <ACPI CPU> on acpi0
p4tcc7: <CPU Frequency Thermal Control> on cpu7
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xccfff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 6 ports with 6 removable, self powered
ugen3.2: <Peppercon AG> at usbus3
ums0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
ums0: 3 buttons and [XYZ] coordinates ID=0
ukbd0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
kbd2 at ukbd0
da0 at twa0 bus 0 target 0 lun 0
da0: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da0: 100.000MB/s transfers
da0: 715245MB (1464821760 512 byte sectors: 255H 63S/T 91180C)
da1 at twa0 bus 0 target 1 lun 0
da1: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da1: 100.000MB/s transfers
da1: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #6 Launched!
da2 at twa0 bus 0 target 2 lun 0
da2: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da2: 100.000MB/s transfers
da2: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da3 at twa0 bus 0 target 3 lun 0
da3: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da3: 100.000MB/s transfers
da3: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da4 at twa0 bus 0 target 4 lun 0
da4: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da4: 100.000MB/s transfers
da4: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da5 at twa0 bus 0 target 5 lun 0
da5: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da5: 100.000MB/s transfers
da5: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da6 at twa0 bus 0 target 6 lun 0
da6: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da6: 100.000MB/s transfers
da6: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da7 at twa0 bus 0 target 7 lun 0
da7: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da7: 100.000MB/s transfers
da7: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da8 at twa0 bus 0 target 8 lun 0
da8: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da8: 100.000MB/s transfers
da8: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da9 at twa0 bus 0 target 9 lun 0
da9: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da9: 100.000MB/s transfers
da9: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da10 at twa0 bus 0 target 10 lun 0
da10: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da10: 100.000MB/s transfers
da10: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da11 at twa0 bus 0 target 11 lun 0
da11: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da11: 100.000MB/s transfers
da11: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da12 at twa0 bus 0 target 12 lun 0
da12: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da12: 100.000MB/s transfers
da12: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da13 at twa0 bus 0 target 13 lun 0
da13: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da13: 100.000MB/s transfers
da13: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da14 at twa0 bus 0 target 14 lun 0
da14: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da14: 100.000MB/s transfers
da14: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da15 at twa0 bus 0 target 15 lun 0
da15: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da15: 100.000MB/s transfers
da15: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da16 at twa0 bus 0 target 16 lun 0
da16: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da16: 100.000MB/s transfers
da16: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da17 at twa0 bus 0 target 17 lun 0
da17: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da17: 100.000MB/s transfers
da17: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da18 at twa0 bus 0 target 18 lun 0
da18: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da18: 100.000MB/s transfers
da18: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da19 at twa0 bus 0 target 19 lun 0
da19: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da19: 100.000MB/s transfers
da19: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da20 at twa0 bus 0 target 20 lun 0
da20: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da20: 100.000MB/s transfers
da20: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da21 at twa0 bus 0 target 21 lun 0
da21: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da21: 100.000MB/s transfers
da21: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
da22 at twa0 bus 0 target 22 lun 0
da22: <AMCC 9650SE-24M DISK 4.10> Fixed Direct Access SCSI-5 device
da22: 100.000MB/s transfers
da22: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C)
Trying to mount root from ufs:/dev/da0s1a
WARNING: / was not properly dismounted
ZFS filesystem version 13
ZFS storage pool version 13
em2: link state changed to UP
cannot do debug because server freeze on dumping
i will recomplie kernel to ddb/kdb and try onlin debuging (if server will not freeze )
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201003311220.o2VCKVG6092630>