Date: Tue, 22 Nov 2011 10:31:44 GMT From: Jacek Kalamarz <jkalamarz@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/162751: [zfs] [panic] kernel panics during file operations Message-ID: <201111221031.pAMAVieS061982@red.freebsd.org> Resent-Message-ID: <201111221040.pAMAe7Gk097820@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 162751
>Category: kern
>Synopsis: [zfs] [panic] kernel panics during file operations
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Nov 22 10:40:06 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator: Jacek Kalamarz
>Release: 8.2-RELEASE-p4
>Organization:
>Environment:
FreeBSD kim.rolskiego.net 8.2-RELEASE-p4 FreeBSD 8.2-RELEASE-p4 #0: Tue Oct 25 10:15:05 CEST 2011 root@kim.rolskiego.net:/usr/obj/usr/src/sys/GENERIC amd64
Celeron 1.2GHz, 2GB RAM, zpool created on 1TB partition
ZFS details:
simson@kim:usr/src/sys/GENERIC$ zpool status
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
ad4s1d ONLINE 0 0 0
errors: No known data errors
simson@kim:usr/src/sys/GENERIC$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 111G 803G 23K none
tank/home 47.7G 52.3G 16.3G /home
tank/storage 54.1G 246G 54.1G /storage
tank/tmp 30.2M 50.0G 30.2M /tmp
tank/usr 2.19G 2.81G 2.19G /usr
tank/var 4.58G 5.42G 4.58G /var
simson@kim:usr/src/sys/GENERIC$ zfs list -H -t snapshot | wc -l
36
>Description:
Since ZFS is used on the machine, the machine crashes about once a week.
Previously (using only UFS2 partitions), the machine was stable for 3 months.
The logs show exactly the same code line each time:
simson@kim:usr/src/sys/GENERIC$ kgdb kernel.debug /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x460700000c9e
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80820f37
stack pointer = 0x28:0xffffff8092280770
frame pointer = 0x28:0xffffff8092280800
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 38 (arc_reclaim_thread)
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff805f4e0e at kdb_backtrace+0x5e
#1 0xffffffff805c2d07 at panic+0x187
#2 0xffffffff808ac630 at trap_fatal+0x291
#3 0xffffffff808aca0f at trap_pfault+0x28f
#4 0xffffffff808aceef at trap+0x3df
#5 0xffffffff80894fe4 at calltrap+0x8
#6 0xffffffff80821932 at vm_page_remove+0x32
#7 0xffffffff80821a7d at vm_page_free_toq+0x6d
#8 0xffffffff8082085b at vm_object_page_remove+0x11b
#9 0xffffffff80818c33 at vm_map_delete+0x313
#10 0xffffffff80818d41 at vm_map_remove+0x51
#11 0xffffffff8080d6a5 at uma_large_free+0x55
#12 0xffffffff805aff97 at free+0x77
#13 0xffffffff80e36351 at arc_buf_destroy+0x101
#14 0xffffffff80e39614 at arc_evict+0x2f4
#15 0xffffffff80e3a6ec at arc_adjust+0x1bc
#16 0xffffffff80e3a9b0 at arc_reclaim_thread+0x1a0
#17 0xffffffff805994f8 at fork_exit+0x118
Uptime: 11d9h8m14s
Physical memory: 1997 MB
Dumping 1632 MB: 1617 1601 1585 1569 1553 1537 1521 1505 1489 1473 1457 1441 1425 1409 1393 1377 1361 1345 1329 1313 1297 1281 1265 1249 1233 1217 1201 1185 1169 1153 1137 1121 1105 1089 1073 1057 1041 1025 1009 993 977 961 945 929 913 897 881 865 849 833 817 801 785 769 753 737 721 705 689 673 657 641 625 609 593 577 561 545 529 513 497 481 465 449 433 417 401 385 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /usr/local/modules/fuse.ko...done.
Loaded symbols for /usr/local/modules/fuse.ko
Reading symbols from /boot/kernel/snp.ko...Reading symbols from /boot/kernel/snp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/snp.ko
#0 doadump () at pcpu.h:224
224 __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) bt
#0 doadump () at pcpu.h:224
#1 0xffffffff805c28be in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419
#2 0xffffffff805c2cf1 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:592
#3 0xffffffff808ac630 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:783
#4 0xffffffff808aca0f in trap_pfault (frame=0xffffff80922806c0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699
#5 0xffffffff808aceef in trap (frame=0xffffff80922806c0) at /usr/src/sys/amd64/amd64/trap.c:449
#6 0xffffffff80894fe4 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224
#7 0xffffffff80820f37 in vm_page_splay (pindex=223004, root=0x460700000c66) at /usr/src/sys/vm/vm_page.c:624
#8 0xffffffff80821932 in vm_page_remove (m=0xffffff007b206c70) at /usr/src/sys/vm/vm_page.c:741
#9 0xffffffff80821a7d in vm_page_free_toq (m=0xffffff007b206c70) at /usr/src/sys/vm/vm_page.c:1562
#10 0xffffffff8082085b in vm_object_page_remove (object=0xffffffff80b957a0, start=222976, end=223008, clean_only=0) at /usr/src/sys/vm/vm_object.c:1788
#11 0xffffffff80818c33 in vm_map_delete (map=0xffffff00010000e8, start=Variable "start" is not available.
) at /usr/src/sys/vm/vm_map.c:2715
#12 0xffffffff80818d41 in vm_map_remove (map=0xffffff00010000e8, start=18446743524867047424, end=18446743524867178496) at /usr/src/sys/vm/vm_map.c:2846
#13 0xffffffff8080d6a5 in uma_large_free (slab=0xffffff00291c2470) at /usr/src/sys/vm/uma_core.c:3084
#14 0xffffffff805aff97 in free (addr=0xffffff8036700000, mtp=0xffffffff80f277c0) at /usr/src/sys/kern/kern_malloc.c:506
#15 0xffffffff80e36351 in arc_buf_destroy (buf=0xffffff002c2710d8, recycle=Variable "recycle" is not available.
) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1497
#16 0xffffffff80e39614 in arc_evict (state=0xffffffff80f11b00, spa=0, bytes=35238618, recycle=0, type=ARC_BUFC_DATA) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1780
#17 0xffffffff80e3a6ec in arc_adjust () at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1993
#18 0xffffffff80e3a9b0 in arc_reclaim_thread (dummy=Variable "dummy" is not available.
) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2251
#19 0xffffffff805994f8 in fork_exit (callout=0xffffffff80e3a810 <arc_reclaim_thread>, arg=0x0, frame=0xffffff8092280c40) at /usr/src/sys/kern/kern_fork.c:845
#20 0xffffffff808954ae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:565
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000001 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0xffffffff80b67d80 in affinity ()
#46 0x0000000000000000 in ?? ()
#47 0x0000000000000000 in ?? ()
#48 0xffffff00019648c0 in ?? ()
#49 0xffffff80922806a0 in ?? ()
#50 0xffffff8092280648 in ?? ()
#51 0xffffff00015c6000 in ?? ()
#52 0xffffffff805e81b9 in sched_switch (td=0xffffffff80e3a810, newtd=0x0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1852
Previous frame inner to this frame (corrupt stack?)
(kgdb) f 7
#7 0xffffffff80820f37 in vm_page_splay (pindex=223004, root=0x460700000c66) at /usr/src/sys/vm/vm_page.c:624
624 lefttreemax->right = root;
(kgdb) l *0xffffffff80820f37
0xffffffff80820f37 is in vm_page_splay (/usr/src/sys/vm/vm_page.c:598).
598 if (pindex < root->pindex) {
(kgdb) l 595,630
595 return (root);
596 lefttreemax = righttreemin = &dummy;
597 for (;; root = y) {
598 if (pindex < root->pindex) {
599 if ((y = root->left) == NULL)
600 break;
601 if (pindex < y->pindex) {
602 /* Rotate right. */
603 root->left = y->right;
604 y->right = root;
605 root = y;
606 if ((y = root->left) == NULL)
607 break;
608 }
609 /* Link into the new root's right tree. */
610 righttreemin->left = root;
611 righttreemin = root;
612 } else if (pindex > root->pindex) {
613 if ((y = root->right) == NULL)
614 break;
615 if (pindex > y->pindex) {
616 /* Rotate left. */
617 root->right = y->left;
618 y->left = root;
619 root = y;
620 if ((y = root->right) == NULL)
621 break;
622 }
623 /* Link into the new root's left tree. */
624 lefttreemax->right = root;
625 lefttreemax = root;
626 } else
627 break;
628 }
629 /* Assemble the new root. */
630 lefttreemax->right = root->left;
(kgdb) p righttreemin
$7 = 0xffffff8092280780
(kgdb) p lefttreemax
$8 = 0xffffff8092280780
(kgdb) p &dummy
$9 = (struct vm_page *) 0xffffff8092280780
(kgdb) p dummy
$10 = {pageq = {tqe_next = 0x0, tqe_prev = 0xffffff007ce2f880}, listq = {tqe_next = 0xffffff8092280850, tqe_prev = 0xffffffff8080ff7b}, left = 0x0, right = 0xffffff007cbb5158, object = 0xffffff007ce2f800,
pindex = 18446742974224550080, phys_addr = 18446742976246783680, md = {pv_list = {tqh_first = 0xffffff007af92548, tqh_last = 0xffffff00019648c0}, pat_mode = 513}, queue = 64 '@', segind = 8 '\b',
flags = 37416, order = 128 '\200', pool = 255 'y', cow = 65535, wire_count = 2156563222, hold_count = -1, oflags = 65535, act_count = 192 'A', busy = 72 'H', valid = 150 '\226', dirty = 1 '\001'}
(kgdb) p y
$11 = 0xffffff007b206c70
(kgdb) p *y
$12 = {pageq = {tqe_next = 0x0, tqe_prev = 0xffffffff80b95ec0}, listq = {tqe_next = 0xffffff00794f2b48, tqe_prev = 0xffffff007b8530c0}, left = 0x0, right = 0xffffff007bead320, object = 0xffffffff80b955a0,
pindex = 223004, phys_addr = 1113141248, md = {pv_list = {tqh_first = 0x0, tqh_last = 0xffffff007b206cb8}, pat_mode = 6}, queue = 0 '\0', segind = 2 '\002', flags = 2048, order = 13 '\r', pool = 0 '\0',
cow = 0, wire_count = 0, hold_count = 0, oflags = 0, act_count = 0 '\0', busy = 0 '\0', valid = 255 'y', dirty = 0 '\0'}
(kgdb) p pindex
$13 = 223004
(kgdb) p root
$14 = 0x460700000c66
(kgdb) p *root
Cannot access memory at address 0x460700000c66
(kgdb)
Other dumps:
vmcore.1:
(kgdb) p righttreemin
$1 = 0xffffff8092280780
(kgdb) p lefttreemax
$2 = 0xffffff8092280780
(kgdb) p y->pindex
$3 = 128084
(kgdb) p pindex
$4 = 128084
(kgdb) p root
$5 = 0x592e00000c66
vmcore.3:
(kgdb) p righttreemin
$1 = 0xffffff8092248780
(kgdb) p lefttreemax
$2 = 0xffffff8092248780
(kgdb) p pindex
$3 = 121576
(kgdb) p y->pindex
$4 = 121576
(kgdb) p root
$5 = 0x2b1600000c66
>How-To-Repeat:
Probably crashes after large load (tar, http serving, etc.)
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201111221031.pAMAVieS061982>