Date: Fri, 21 Mar 2014 13:31:08 +0600 From: "Eugene M. Zheganin" <emz@norma.perm.ru> To: freebsd-stable <freebsd-stable@FreeBSD.org> Subject: crash on zpool import - help get data back Message-ID: <532BEABC.5050808@norma.perm.ru>
next in thread | raw e-mail | index | archive | help
Hi. After some time using zfs on a 10.x server (and a couple of panics) I'm now getting the reproducible and repeatable panic on any operations involving particular pool. I managed to boot from USB stick, import only healthy pools and substitite the server's zpool.cache file with one referencing only healthy pools. Now I'm able to boot, but when I try to import the remaining pool I'm getting the panic (attached below). Now questions: - do I understand correctly that "#7 0xffffffff8188e076 in vdev_readable (vd=0x0)" means vd is NULL, and this is triggering the panic ? - I saw a similar (but not identical) panic in http://lists.freebsd.org/pipermail/freebsd-fs/2012-January/013513.html , - are there any possible tricks that could help me getting my data ? the target pool itself shows as healthy too: # zpool import pool: xtank id: 16620000996171732653 state: ONLINE status: Some supported features are not enabled on the pool. action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: xtank ONLINE mirror-0 ONLINE gpt/xtank0 ONLINE gpt/xtank1 ONLINE logs mirror-1 ONLINE gpt/xlog0 ONLINE gpt/xlog1 ONLINE Backtrace and stuff: ===Cut=== # less core.txt.4 witchdoctor.hq.norma.perm.ru dumped core - see /var/crash/vmcore.4 Fri Mar 21 13:00:30 YEKT 2014 FreeBSD witchdoctor.hq.norma.perm.ru 10.0-STABLE FreeBSD 10.0-STABLE #0 r263266: Mon Mar 17 23:17:32 YEKT 2014 emz@witchdoctor.hq.norma.perm.ru:/usr/obj/usr/src/sys/GENERIC amd64 panic: page fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0x50 cpuid = 1; apic id = 01 fault code = supervisor read data, page not present fault virtual address = 0xa0 instruction pointer = 0x20:0xffffffff8188e076 fault code = supervisor read data, page not present stack pointer = 0x28:0xfffffe01208e5b00 instruction pointer = 0x20:0xffffffff818ab666 frame pointer = 0x28:0xfffffe01208e5b10 stack pointer = 0x28:0xfffffe0120fc85b0 code segment = base 0x0, limit 0xfffff, type 0x1b frame pointer = 0x28:0xfffffe0120fc8640 = DPL 0, pres 1, long 1, def32 0, gran 1 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, processor eflags = resume, IOPL = 0 interrupt enabled, resume, IOPL = 0 current process = 0 (system_taskq_2) trap number = 12 current process = 1363 (zpool) panic: page fault cpuid = 3 KDB: stack backtrace: #0 0xffffffff808f01d0 at kdb_backtrace+0x60 #1 0xffffffff808b7ba5 at panic+0x155 #2 0xffffffff80c98f32 at trap_fatal+0x3a2 #3 0xffffffff80c99209 at trap_pfault+0x2c9 #4 0xffffffff80c9899b at trap+0x5bb #5 0xffffffff80c7fc52 at calltrap+0x8 #6 0xffffffff81894740 at vdev_mirror_child_select+0x70 #7 0xffffffff81894284 at vdev_mirror_io_start+0x234 #8 0xffffffff818ae754 at zio_vdev_io_start+0x184 #9 0xffffffff818aba8a at zio_execute+0x15a #10 0xffffffff8183a158 at arc_read+0x958 #11 0xffffffff81852fde at traverse_prefetcher+0x13e #12 0xffffffff8185243d at traverse_visitbp+0x20d #13 0xffffffff81852e6f at traverse_dnode+0xef #14 0xffffffff81852bb7 at traverse_visitbp+0x987 #15 0xffffffff81852613 at traverse_visitbp+0x3e3 #16 0xffffffff81852613 at traverse_visitbp+0x3e3 #17 0xffffffff81852613 at traverse_visitbp+0x3e3 Uptime: 6m45s Dumping 326 out of 4043 MB:..5%..15%..25%..35%..45%..54%..64%..74%..84%..94% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done. Loaded symbols for /boot/kernel/geom_mirror.ko.symbols Reading symbols from /boot/kernel/ums.ko.symbols...done. Loaded symbols for /boot/kernel/ums.ko.symbols Reading symbols from /boot/kernel/uhid.ko.symbols...done. Loaded symbols for /boot/kernel/uhid.ko.symbols #0 doadump (textdump=<value optimized out>) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=<value optimized out>) at pcpu.h:219 #1 0xffffffff808b7820 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0xffffffff808b7be4 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80c98f32 in trap_fatal (frame=<value optimized out>, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:875 #4 0xffffffff80c99209 in trap_pfault (frame=0xfffffe01208e5a50, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:692 #5 0xffffffff80c9899b in trap (frame=0xfffffe01208e5a50) at /usr/src/sys/amd64/amd64/trap.c:456 #6 0xffffffff80c7fc52 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #7 0xffffffff8188e076 in vdev_readable (vd=0x0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:2632 #8 0xffffffff81894740 in vdev_mirror_child_select (zio=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:238 #9 0xffffffff81894284 in vdev_mirror_io_start (zio=0xfffff8004f228398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:295 #10 0xffffffff818ae754 in zio_vdev_io_start (zio=0xfffff8004f228398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2514 #11 0xffffffff818aba8a in zio_execute (zio=0xfffff8004f228398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1346 #12 0xffffffff8183a158 in arc_read (pio=0x0, spa=0xfffff800799b8000, bp=<value optimized out>, done=0x2, private=0x0, priority=ZIO_PRIORITY_ASYNC_READ, zio_flags=512, arc_flags=<value optimized out>, zb=0x0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3417 #13 0xffffffff81852fde in traverse_prefetcher (spa=0xfffff800799b8000, zilog=0xf01ff, bp=<value optimized out>, zb=<value optimized out>, dnp=0xfffff8004f1d3240, arg=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:451 #14 0xffffffff8185243d in traverse_visitbp (td=0xfffffe01208e6980, dnp=0xfffffe00276fe800, bp=0xfffffe00276fe980, zb=0xfffffe01208e5f08) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:250 #15 0xffffffff81852e6f in traverse_dnode (td=0xfffffe01208e6980, dnp=0xfffffe00276fe800, objset=41, object=57388) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:417 #16 0xffffffff81852bb7 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0xfffffe00276fd000, bp=0xfffffe00276f1080, zb=0xfffffe01208e6128) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:309 #17 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0xfffff80079bb9800, bp=0xfffffe0027709700, zb=0xfffffe01208e6258) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #18 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0xfffff80079bb9800, bp=0xfffffe00275dd000, zb=0xfffffe01208e6388) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #19 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0xfffff80079bb9800, bp=0xfffffe0027651000, zb=0xfffffe01208e64b8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #20 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0xfffff80079bb9800, bp=0xfffffe0027584000, zb=0xfffffe01208e65e8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #21 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0xfffff80079bb9800, bp=0xfffffe002755c000, zb=0xfffffe01208e6718) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #22 0xffffffff81852613 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0xfffff80079bb9800, bp=0xfffff80079bb9840, zb=0xfffffe01208e67d8) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:284 #23 0xffffffff81852e04 in traverse_dnode (td=0xfffffe01208e6980, dnp=0xfffff80079bb9800, objset=41, object=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:407 #24 0xffffffff818528c0 in traverse_visitbp (td=0xfffffe01208e6980, dnp=0x0, bp=0xfffff800794c6280, zb=0xfffffe01208e6960) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:338 #25 0xffffffff818521d6 in traverse_prefetch_thread (arg=0xfffffe0120fc9420) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:470 #26 0xffffffff81828c00 in taskq_run (arg=0xfffff80018b01ea0, pending=983551) at /usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_taskq.c:109 #27 0xffffffff808fe1b6 in taskqueue_run_locked (queue=0xfffff8001614c300) at /usr/src/sys/kern/subr_taskqueue.c:342 #28 0xffffffff808fec28 in taskqueue_thread_loop (arg=<value optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:563 #29 0xffffffff80889cba in fork_exit ( callout=0xffffffff808feb80 <taskqueue_thread_loop>, arg=0xfffff8001616f720, frame=0xfffffe01208e6ac0) at /usr/src/sys/kern/kern_fork.c:995 #30 0xffffffff80c8018e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #31 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) ===Cut=== Thanks. Eugene.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?532BEABC.5050808>