Date: Tue, 10 Jun 2014 11:58:08 -0400 From: John Baldwin <jhb@freebsd.org> To: freebsd-fs@freebsd.org Cc: grarpamp <grarpamp@gmail.com> Subject: Re: ZFS import panic (kgdb backtrace attached) Message-ID: <201406101158.08599.jhb@freebsd.org> In-Reply-To: <CAD2Ti29gKmED34S5z6NEUnaGOsx8m2uPEJiPWPZLcebJ6PD-mw@mail.gmail.com> References: <CAD2Ti29gKmED34S5z6NEUnaGOsx8m2uPEJiPWPZLcebJ6PD-mw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday, June 09, 2014 2:37:36 pm grarpamp wrote:
> ZFS pool was 96% full and under heavy sequential write, and panicked.
> Dumps were not enabled so this first panic was lost. Fixed that, then...
>
> # zpool import -o readonly=on -f pool
> (ok, zpool export pool)
> # zpool import -f pool
> (repeatably panics, coredump, reboot)
>
> FreeBSD 8.4-STABLE #0 r265935 i386
>
> [/usr/include]% kgdb /boot/kernel/kernel /.../vmcore.1
> GNU gdb 6.1.1 [FreeBSD]
> This GDB was configured as "i386-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0x11
> fault code = supervisor read, page not present
> instruction pointer = 0x20:0xc13cb9f4
> stack pointer = 0x28:0xfcfb5ac0
> frame pointer = 0x28:0xfcfb5ae4
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, def32 1, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 8 (txg_thread_enter)
> trap number = 12
> panic: page fault
> cpuid = 0
> KDB: stack backtrace:
> #0 0xc094cd8f at kdb_backtrace+0x4f
> #1 0xc091c5bc at panic+0x15c
> #2 0xc0d75193 at trap_fatal+0x323
> #3 0xc0d7529c at trap_pfault+0xfc
> #4 0xc0d7600a at trap+0x44a
> #5 0xc0d5c2dc at calltrap+0x6
> #6 0xc13c91d9 at metaslab_sync+0x509
> #7 0xc13eb280 at vdev_sync+0x90
> #8 0xc13dded6 at spa_sync+0x496
> #9 0xc13e8835 at txg_sync_thread+0x145
> #10 0xc08ef767 at fork_exit+0x97
> #11 0xc0d5c354 at fork_trampoline+0x8
> Uptime: 17m21s
> Physical memory: 2026 MB
> Dumping 162 MB: 147 131 115 99 83 67 51 35 19 3
>
> Loaded symbols for /boot/kernel/zfs.ko
> Loaded symbols for /boot/kernel/opensolaris.ko
> Loaded symbols for /boot/kernel/geom_eli.ko
> Loaded symbols for /boot/kernel/crypto.ko
> Loaded symbols for /boot/kernel/zlib.ko
> Loaded symbols for /boot/kernel/snd_ich.ko
> Loaded symbols for /boot/kernel/sound.ko
> Loaded symbols for /boot/kernel/drm.ko
> Loaded symbols for /boot/kernel/i915.ko
> Loaded symbols for /boot/kernel/atapicam.ko
> Loaded symbols for /boot/kernel/cpuctl.ko
>
> #0 doadump () at pcpu.h:244
> 244 __asm("movl %%fs:0,%0" : "=r" (td));
>
> (kgdb) bt
> #0 doadump () at pcpu.h:244
> #1 0xc091c313 in boot (howto=260) at /.../src/sys/kern/kern_shutdown.c:443
> #2 0xc091c5fe in panic (fmt=<value optimized out>) at
> /.../src/sys/kern/kern_shutdown.c:634
> #3 0xc0d75193 in trap_fatal (frame=0xfcfb5a80, eva=17) at
> /.../src/sys/i386/i386/trap.c:1010
> #4 0xc0d7529c in trap_pfault (frame=0xfcfb5a80, usermode=0, eva=17)
> at /.../src/sys/i386/i386/trap.c:872
> #5 0xc0d7600a in trap (frame=0xfcfb5a80) at
/.../src/sys/i386/i386/trap.c:546
> #6 0xc0d5c2dc in calltrap () at /.../src/sys/i386/i386/exception.s:168
> #7 0xc13cb9f4 in range_tree_vacate (rt=0xc83dc000, func=0, arg=0x0)
> at
/.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c:364
> #8 0xc13c91d9 in metaslab_sync (msp=0xc8309000, txg=21088308)
> at
/.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1486
> #9 0xc13eb280 in vdev_sync (vd=0xc7f69800, txg=Unhandled dwarf
> expression opcode 0x93
> ) at
/.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:2274
> #10 0xc13dded6 in spa_sync (spa=0xdf3bf000, txg=21088308)
> at
/.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6506
> #11 0xc13e8835 in txg_sync_thread (arg=0xc7907400) at
>
/.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:518
> #12 0xc08ef767 in fork_exit (callout=0xc13e86f0 <txg_sync_thread>,
> arg=0xc7907400, frame=0xfcfb5d28) at /.../src/sys/kern/kern_fork.c:872
> #13 0xc0d5c354 in fork_trampoline () at
/.../src/sys/i386/i386/exception.s:275
>
> (kgdb) list *0xc13cb9f4
> 0xc13cb9f4 is in range_tree_vacate
>
(/.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c:364).
> 359 void *cookie = NULL;
> 360
> 361 ASSERT(MUTEX_HELD(rt->rt_lock));
> 362
> 363 if (rt->rt_ops != NULL)
> 364 rt->rt_ops->rtop_vacate(rt, rt->rt_arg);
> 365
> 366 while ((rs = avl_destroy_nodes(&rt->rt_root, &cookie))
> != NULL) {
> 367 if (func != NULL)
> 368 func(arg, rs->rs_start, rs->rs_end -
> rs->rs_start);
Can you do 'frame 7' and 'p *rt' and 'p *rt->rt_ops'?
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201406101158.08599.jhb>