From owner-freebsd-bugs@FreeBSD.ORG Tue Nov 22 10:40:12 2011 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEBF7106566B for ; Tue, 22 Nov 2011 10:40:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5720D8FC13 for ; Tue, 22 Nov 2011 10:40:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id pAMAe7nj097821 for ; Tue, 22 Nov 2011 10:40:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id pAMAe7Gk097820; Tue, 22 Nov 2011 10:40:07 GMT (envelope-from gnats) Resent-Date: Tue, 22 Nov 2011 10:40:07 GMT Resent-Message-Id: <201111221040.pAMAe7Gk097820@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Jacek Kalamarz Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CEF5F106564A for ; Tue, 22 Nov 2011 10:31:44 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id BDFA68FC0A for ; Tue, 22 Nov 2011 10:31:44 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id pAMAVi5n061983 for ; Tue, 22 Nov 2011 10:31:44 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id pAMAVieS061982; Tue, 22 Nov 2011 10:31:44 GMT (envelope-from nobody) Message-Id: <201111221031.pAMAVieS061982@red.freebsd.org> Date: Tue, 22 Nov 2011 10:31:44 GMT From: Jacek Kalamarz To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/162751: [zfs] [panic] kernel panics during file operations X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2011 10:40:12 -0000 >Number: 162751 >Category: kern >Synopsis: [zfs] [panic] kernel panics during file operations >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Nov 22 10:40:06 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Jacek Kalamarz >Release: 8.2-RELEASE-p4 >Organization: >Environment: FreeBSD kim.rolskiego.net 8.2-RELEASE-p4 FreeBSD 8.2-RELEASE-p4 #0: Tue Oct 25 10:15:05 CEST 2011 root@kim.rolskiego.net:/usr/obj/usr/src/sys/GENERIC amd64 Celeron 1.2GHz, 2GB RAM, zpool created on 1TB partition ZFS details: simson@kim:usr/src/sys/GENERIC$ zpool status pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 ad4s1d ONLINE 0 0 0 errors: No known data errors simson@kim:usr/src/sys/GENERIC$ zfs list NAME USED AVAIL REFER MOUNTPOINT tank 111G 803G 23K none tank/home 47.7G 52.3G 16.3G /home tank/storage 54.1G 246G 54.1G /storage tank/tmp 30.2M 50.0G 30.2M /tmp tank/usr 2.19G 2.81G 2.19G /usr tank/var 4.58G 5.42G 4.58G /var simson@kim:usr/src/sys/GENERIC$ zfs list -H -t snapshot | wc -l 36 >Description: Since ZFS is used on the machine, the machine crashes about once a week. Previously (using only UFS2 partitions), the machine was stable for 3 months. The logs show exactly the same code line each time: simson@kim:usr/src/sys/GENERIC$ kgdb kernel.debug /var/crash/vmcore.0 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x460700000c9e fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80820f37 stack pointer = 0x28:0xffffff8092280770 frame pointer = 0x28:0xffffff8092280800 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 38 (arc_reclaim_thread) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xffffffff805f4e0e at kdb_backtrace+0x5e #1 0xffffffff805c2d07 at panic+0x187 #2 0xffffffff808ac630 at trap_fatal+0x291 #3 0xffffffff808aca0f at trap_pfault+0x28f #4 0xffffffff808aceef at trap+0x3df #5 0xffffffff80894fe4 at calltrap+0x8 #6 0xffffffff80821932 at vm_page_remove+0x32 #7 0xffffffff80821a7d at vm_page_free_toq+0x6d #8 0xffffffff8082085b at vm_object_page_remove+0x11b #9 0xffffffff80818c33 at vm_map_delete+0x313 #10 0xffffffff80818d41 at vm_map_remove+0x51 #11 0xffffffff8080d6a5 at uma_large_free+0x55 #12 0xffffffff805aff97 at free+0x77 #13 0xffffffff80e36351 at arc_buf_destroy+0x101 #14 0xffffffff80e39614 at arc_evict+0x2f4 #15 0xffffffff80e3a6ec at arc_adjust+0x1bc #16 0xffffffff80e3a9b0 at arc_reclaim_thread+0x1a0 #17 0xffffffff805994f8 at fork_exit+0x118 Uptime: 11d9h8m14s Physical memory: 1997 MB Dumping 1632 MB: 1617 1601 1585 1569 1553 1537 1521 1505 1489 1473 1457 1441 1425 1409 1393 1377 1361 1345 1329 1313 1297 1281 1265 1249 1233 1217 1201 1185 1169 1153 1137 1121 1105 1089 1073 1057 1041 1025 1009 993 977 961 945 929 913 897 881 865 849 833 817 801 785 769 753 737 721 705 689 673 657 641 625 609 593 577 561 545 529 513 497 481 465 449 433 417 401 385 369 353 337 321 305 289 273 257 241 225 209 193 177 161 145 129 113 97 81 65 49 33 17 1 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /usr/local/modules/fuse.ko...done. Loaded symbols for /usr/local/modules/fuse.ko Reading symbols from /boot/kernel/snp.ko...Reading symbols from /boot/kernel/snp.ko.symbols...done. done. Loaded symbols for /boot/kernel/snp.ko #0 doadump () at pcpu.h:224 224 __asm("movq %%gs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:224 #1 0xffffffff805c28be in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:419 #2 0xffffffff805c2cf1 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:592 #3 0xffffffff808ac630 in trap_fatal (frame=0xc, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:783 #4 0xffffffff808aca0f in trap_pfault (frame=0xffffff80922806c0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699 #5 0xffffffff808aceef in trap (frame=0xffffff80922806c0) at /usr/src/sys/amd64/amd64/trap.c:449 #6 0xffffffff80894fe4 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #7 0xffffffff80820f37 in vm_page_splay (pindex=223004, root=0x460700000c66) at /usr/src/sys/vm/vm_page.c:624 #8 0xffffffff80821932 in vm_page_remove (m=0xffffff007b206c70) at /usr/src/sys/vm/vm_page.c:741 #9 0xffffffff80821a7d in vm_page_free_toq (m=0xffffff007b206c70) at /usr/src/sys/vm/vm_page.c:1562 #10 0xffffffff8082085b in vm_object_page_remove (object=0xffffffff80b957a0, start=222976, end=223008, clean_only=0) at /usr/src/sys/vm/vm_object.c:1788 #11 0xffffffff80818c33 in vm_map_delete (map=0xffffff00010000e8, start=Variable "start" is not available. ) at /usr/src/sys/vm/vm_map.c:2715 #12 0xffffffff80818d41 in vm_map_remove (map=0xffffff00010000e8, start=18446743524867047424, end=18446743524867178496) at /usr/src/sys/vm/vm_map.c:2846 #13 0xffffffff8080d6a5 in uma_large_free (slab=0xffffff00291c2470) at /usr/src/sys/vm/uma_core.c:3084 #14 0xffffffff805aff97 in free (addr=0xffffff8036700000, mtp=0xffffffff80f277c0) at /usr/src/sys/kern/kern_malloc.c:506 #15 0xffffffff80e36351 in arc_buf_destroy (buf=0xffffff002c2710d8, recycle=Variable "recycle" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1497 #16 0xffffffff80e39614 in arc_evict (state=0xffffffff80f11b00, spa=0, bytes=35238618, recycle=0, type=ARC_BUFC_DATA) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1780 #17 0xffffffff80e3a6ec in arc_adjust () at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1993 #18 0xffffffff80e3a9b0 in arc_reclaim_thread (dummy=Variable "dummy" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2251 #19 0xffffffff805994f8 in fork_exit (callout=0xffffffff80e3a810 , arg=0x0, frame=0xffffff8092280c40) at /usr/src/sys/kern/kern_fork.c:845 #20 0xffffffff808954ae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:565 #21 0x0000000000000000 in ?? () #22 0x0000000000000000 in ?? () #23 0x0000000000000001 in ?? () #24 0x0000000000000000 in ?? () #25 0x0000000000000000 in ?? () #26 0x0000000000000000 in ?? () #27 0x0000000000000000 in ?? () #28 0x0000000000000000 in ?? () #29 0x0000000000000000 in ?? () #30 0x0000000000000000 in ?? () #31 0x0000000000000000 in ?? () #32 0x0000000000000000 in ?? () #33 0x0000000000000000 in ?? () #34 0x0000000000000000 in ?? () #35 0x0000000000000000 in ?? () #36 0x0000000000000000 in ?? () #37 0x0000000000000000 in ?? () #38 0x0000000000000000 in ?? () #39 0x0000000000000000 in ?? () #40 0x0000000000000000 in ?? () #41 0x0000000000000000 in ?? () #42 0x0000000000000000 in ?? () #43 0x0000000000000000 in ?? () #44 0x0000000000000000 in ?? () #45 0xffffffff80b67d80 in affinity () #46 0x0000000000000000 in ?? () #47 0x0000000000000000 in ?? () #48 0xffffff00019648c0 in ?? () #49 0xffffff80922806a0 in ?? () #50 0xffffff8092280648 in ?? () #51 0xffffff00015c6000 in ?? () #52 0xffffffff805e81b9 in sched_switch (td=0xffffffff80e3a810, newtd=0x0, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1852 Previous frame inner to this frame (corrupt stack?) (kgdb) f 7 #7 0xffffffff80820f37 in vm_page_splay (pindex=223004, root=0x460700000c66) at /usr/src/sys/vm/vm_page.c:624 624 lefttreemax->right = root; (kgdb) l *0xffffffff80820f37 0xffffffff80820f37 is in vm_page_splay (/usr/src/sys/vm/vm_page.c:598). 598 if (pindex < root->pindex) { (kgdb) l 595,630 595 return (root); 596 lefttreemax = righttreemin = &dummy; 597 for (;; root = y) { 598 if (pindex < root->pindex) { 599 if ((y = root->left) == NULL) 600 break; 601 if (pindex < y->pindex) { 602 /* Rotate right. */ 603 root->left = y->right; 604 y->right = root; 605 root = y; 606 if ((y = root->left) == NULL) 607 break; 608 } 609 /* Link into the new root's right tree. */ 610 righttreemin->left = root; 611 righttreemin = root; 612 } else if (pindex > root->pindex) { 613 if ((y = root->right) == NULL) 614 break; 615 if (pindex > y->pindex) { 616 /* Rotate left. */ 617 root->right = y->left; 618 y->left = root; 619 root = y; 620 if ((y = root->right) == NULL) 621 break; 622 } 623 /* Link into the new root's left tree. */ 624 lefttreemax->right = root; 625 lefttreemax = root; 626 } else 627 break; 628 } 629 /* Assemble the new root. */ 630 lefttreemax->right = root->left; (kgdb) p righttreemin $7 = 0xffffff8092280780 (kgdb) p lefttreemax $8 = 0xffffff8092280780 (kgdb) p &dummy $9 = (struct vm_page *) 0xffffff8092280780 (kgdb) p dummy $10 = {pageq = {tqe_next = 0x0, tqe_prev = 0xffffff007ce2f880}, listq = {tqe_next = 0xffffff8092280850, tqe_prev = 0xffffffff8080ff7b}, left = 0x0, right = 0xffffff007cbb5158, object = 0xffffff007ce2f800, pindex = 18446742974224550080, phys_addr = 18446742976246783680, md = {pv_list = {tqh_first = 0xffffff007af92548, tqh_last = 0xffffff00019648c0}, pat_mode = 513}, queue = 64 '@', segind = 8 '\b', flags = 37416, order = 128 '\200', pool = 255 'y', cow = 65535, wire_count = 2156563222, hold_count = -1, oflags = 65535, act_count = 192 'A', busy = 72 'H', valid = 150 '\226', dirty = 1 '\001'} (kgdb) p y $11 = 0xffffff007b206c70 (kgdb) p *y $12 = {pageq = {tqe_next = 0x0, tqe_prev = 0xffffffff80b95ec0}, listq = {tqe_next = 0xffffff00794f2b48, tqe_prev = 0xffffff007b8530c0}, left = 0x0, right = 0xffffff007bead320, object = 0xffffffff80b955a0, pindex = 223004, phys_addr = 1113141248, md = {pv_list = {tqh_first = 0x0, tqh_last = 0xffffff007b206cb8}, pat_mode = 6}, queue = 0 '\0', segind = 2 '\002', flags = 2048, order = 13 '\r', pool = 0 '\0', cow = 0, wire_count = 0, hold_count = 0, oflags = 0, act_count = 0 '\0', busy = 0 '\0', valid = 255 'y', dirty = 0 '\0'} (kgdb) p pindex $13 = 223004 (kgdb) p root $14 = 0x460700000c66 (kgdb) p *root Cannot access memory at address 0x460700000c66 (kgdb) Other dumps: vmcore.1: (kgdb) p righttreemin $1 = 0xffffff8092280780 (kgdb) p lefttreemax $2 = 0xffffff8092280780 (kgdb) p y->pindex $3 = 128084 (kgdb) p pindex $4 = 128084 (kgdb) p root $5 = 0x592e00000c66 vmcore.3: (kgdb) p righttreemin $1 = 0xffffff8092248780 (kgdb) p lefttreemax $2 = 0xffffff8092248780 (kgdb) p pindex $3 = 121576 (kgdb) p y->pindex $4 = 121576 (kgdb) p root $5 = 0x2b1600000c66 >How-To-Repeat: Probably crashes after large load (tar, http serving, etc.) >Fix: >Release-Note: >Audit-Trail: >Unformatted: