From owner-freebsd-current@FreeBSD.ORG Wed Oct 2 17:59:24 2013 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8192C3F1; Wed, 2 Oct 2013 17:59:24 +0000 (UTC) (envelope-from kwhite@site.uottawa.ca) Received: from courriel.site.uottawa.ca (eecsmail.engineering.uottawa.ca [137.122.24.224]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2578F2CFC; Wed, 2 Oct 2013 17:59:23 +0000 (UTC) Received: from [10.0.2.15] (dsl-74-51-61-7.vianet.ca [74.51.61.7]) (authenticated bits=0) by courriel.site.uottawa.ca (8.14.5/8.14.4) with ESMTP id r92HxLiF027974 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 2 Oct 2013 13:59:22 -0400 (EDT) (envelope-from kwhite@site.uottawa.ca) Date: Wed, 2 Oct 2013 13:59:26 -0400 (EDT) From: Keith White X-X-Sender: kwhite@localhost.my.domain To: Andriy Gapon Subject: Re: ZFS panic with r255937 In-Reply-To: <524C0DB2.5040202@FreeBSD.org> Message-ID: References: <60850.74.51.61.7.1380496284.squirrel@courriel.site.uottawa.ca> <524C0DB2.5040202@FreeBSD.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-current@FreeBSD.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Oct 2013 17:59:24 -0000 On Wed, 2 Oct 2013, Andriy Gapon wrote: > on 30/09/2013 02:11 kwhite@site.uottawa.ca said the following: >> Sorry, debugging this is *way* beyond me. Any hints, patches to try? > > Please share the stack trace. > > -- > Andriy Gapon There's now a pr for this panic: kern/182570 Here's the stack trace: root@freebsd10:/usr/src # kgdb /boot/kernel/kernel /var/crash/vmcore.last GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: solaris assert: dn->dn_maxblkid == 0 && (BP_IS_HOLE(&dn->dn_phys->dn_blkptr[0]) || dnode_block_freed(dn, 0)), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c, line: 598 cpuid = 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00992b3280 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00992b3330 vpanic() at vpanic+0x126/frame 0xfffffe00992b3370 panic() at panic+0x43/frame 0xfffffe00992b33d0 assfail() at assfail+0x22/frame 0xfffffe00992b33e0 dnode_reallocate() at dnode_reallocate+0x225/frame 0xfffffe00992b3430 dmu_object_reclaim() at dmu_object_reclaim+0x123/frame 0xfffffe00992b3480 dmu_recv_stream() at dmu_recv_stream+0xd79/frame 0xfffffe00992b36b0 zfs_ioc_recv() at zfs_ioc_recv+0x96c/frame 0xfffffe00992b3920 zfsdev_ioctl() at zfsdev_ioctl+0x54a/frame 0xfffffe00992b39c0 devfs_ioctl_f() at devfs_ioctl_f+0xf0/frame 0xfffffe00992b3a20 kern_ioctl() at kern_ioctl+0x2ca/frame 0xfffffe00992b3a90 sys_ioctl() at sys_ioctl+0x11f/frame 0xfffffe00992b3ae0 amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe00992b3bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00992b3bf0 --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019dc9ca, rsp = 0x7fffffff5ad8, rbp = 0x7fffffff5b60 --- KDB: enter: panic Uptime: 37m10s Dumping 874 out of 2023 MB: ... (kgdb) where #0 doadump (textdump=1) at pcpu.h:218 #1 0xffffffff808b7217 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xffffffff808b7725 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xffffffff808b7773 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:683 #4 0xffffffff81e52222 in assfail (a=, f=, l=) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 #5 0xffffffff81d09735 in dnode_reallocate (dn=0xfffff8006dde3000, ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024, bonustype=DMU_OT_SA, bonuslen=168, tx=0xfffff8006d7a2600) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:596 #6 0xffffffff81cff463 in dmu_object_reclaim (os=, object=, ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024, bonustype=DMU_OT_SA, bonuslen=168) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c:155 #7 0xffffffff81cfd849 in dmu_recv_stream (drc=0xfffffe00992b3728, fp=, voffp=0xfffffe00992b3718, cleanup_fd=8, action_handlep=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:1231 #8 0xffffffff81d8b1fc in zfs_ioc_recv (zc=0xfffffe00372e1000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4102 #9 0xffffffff81d864ea in zfsdev_ioctl (dev=, zcmd=, arg=, flag=, td=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:5960 #10 0xffffffff807af840 in devfs_ioctl_f (fp=0xfffff800052e1460, com=3222821403, data=0xfffff8006ff1faa0, cred=, td=0xfffff8003f7ba920) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #11 0xffffffff8090e94a in kern_ioctl (td=0xfffff8003f7ba920, fd=, com=0) at file.h:319 #12 0xffffffff8090e62f in sys_ioctl (td=0xfffff8003f7ba920, uap=0xfffffe00992b3b80) at /usr/src/sys/kern/sys_generic.c:698 #13 0xffffffff80caee35 in amd64_syscall (td=0xfffff8003f7ba920, traced=0) at subr_syscall.c:134 #14 0xffffffff80c961ab in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391 #15 0x00000008019dc9ca in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) Here's a how to repeat: Assuming pool "tank" and non-existent tank/nobj tank/xobj === cut here === #!/bin/sh zfs create tank/nobj zfs snapshot tank/nobj@0 __MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6 buildkernel zfs snapshot tank/nobj@1 #find /tank/nobj -name '.h' -print | xargs rm zfs snapshot tank/nobj@2 #rm -rf /tank/nobj/* zfs snapshot tank/nobj@3 __MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6 buildkernel zfs snapshot tank/nobj@4 #rm -rf /tank/nobj/* zfs snapshot tank/nobj@5 zfs send -vR tank/nobj@5 | zfs recv -vF tank/xobj === cut here === The commented out lines are historical attempts. ...keith