From owner-freebsd-current@FreeBSD.ORG Sun Sep 7 15:24:14 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B1F10295 for ; Sun, 7 Sep 2014 15:24:14 +0000 (UTC) Received: from smtprelay02.ispgateway.de (smtprelay02.ispgateway.de [80.67.31.36]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 449A5127F for ; Sun, 7 Sep 2014 15:24:13 +0000 (UTC) Received: from [87.79.252.204] (helo=fabiankeil.de) by smtprelay02.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1XQeJx-0006rI-3g for freebsd-current@freebsd.org; Sun, 07 Sep 2014 17:24:05 +0200 Date: Sun, 7 Sep 2014 17:23:16 +0200 From: Fabian Keil To: freebsd-current@freebsd.org Subject: Re: ZFS-related panic: "possible" spa->spa_errlog_lock deadlock Message-ID: <4fa875ba.3cc970d7@fabiankeil.de> In-Reply-To: <540C66AC.8070809@delphij.net> References: <492dbacb.5942cc9b@fabiankeil.de> <540C66AC.8070809@delphij.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/WvIg7bSSG29Dx92rRK+fj0F"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 15:24:14 -0000 --Sig_/WvIg7bSSG29Dx92rRK+fj0F Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Xin Li wrote: > On 9/7/14 9:02 PM, Fabian Keil wrote: > > Using a kernel built from FreeBSD 11.0-CURRENT r271182 I got the > > following panic yesterday: > >=20 > > [...] Unread portion of the kernel message buffer: [6880] panic: > > deadlkres: possible deadlock detected for 0xfffff80015289490, > > blocked for 1800503 ticks >=20 > Any chance to get all backtraces (e.g. thread apply all bt full 16)? > I think a different thread that held the lock have been blocked, > probably related to your disconnected vdev. Output of "thread apply all bt full 16" is available at: http://www.fabiankeil.de/tmp/freebsd/kgdb-output-spa_errlog_lock-deadlock.t= xt A lot of the backtraces prematurely end with "Cannot access memory at addre= ss", therefore I also added "thread apply all bt" output. Apparently there are at least two additional threads blocking below spa_get= _stats(): Thread 1182 (Thread 101989): #0 sched_switch (td=3D0xfffff800628cc490, newtd=3D, f= lags=3D) at /usr/src/sys/kern/sched_ule.c:1932 #1 0xffffffff805a23c1 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/src/= sys/kern/kern_synch.c:493 #2 0xffffffff805e4bca in sleepq_wait (wchan=3D0x0, pri=3D0) at /usr/src/sy= s/kern/subr_sleepqueue.c:631 #3 0xffffffff80539f10 in _cv_wait (cvp=3D0xfffff80025534a50, lock=3D0xffff= f80025534a30) at /usr/src/sys/kern/kern_condvar.c:139 #4 0xffffffff811721db in zio_wait (zio=3D) at /usr/sr= c/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1442 #5 0xffffffff81102111 in dbuf_read (db=3D, zio=3D, flags=3D) at /usr/src/sys/cddl/cont= rib/opensolaris/uts/common/fs/zfs/dbuf.c:649 #6 0xffffffff81108e6d in dmu_buf_hold (os=3D, object= =3D, offset=3D, tag=3D0x0, dbp=3D= 0xfffffe00955c6648, flags=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:172 #7 0xffffffff81163986 in zap_lockdir (os=3D0xfffff8002b7ab000, obj=3D92, t= x=3D0x0, lti=3DRW_READER, fatreader=3D1, adding=3D0, zapp=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:= 467 #8 0xffffffff811644ad in zap_count (os=3D0x0, zapobj=3D0, count=3D0xfffffe= 00955c66d8) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_= micro.c:712 #9 0xffffffff8114a6dc in spa_get_errlog_size (spa=3D0xfffff800062ed000) at= /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_errlog.c:149 ---Type to continue, or q to quit--- #10 0xffffffff8113f549 in spa_get_stats (name=3D0xfffffe0044cac000 "spacelo= op", config=3D0xfffffe00955c68e8, altroot=3D0xfffffe0044cac430 "", buflen= =3D2048) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3287 #11 0xffffffff81189a45 in zfs_ioc_pool_stats (zc=3D0xfffffe0044cac000) at /= usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1656 #12 0xffffffff81187290 in zfsdev_ioctl (dev=3D, zcmd= =3D, arg=3D, flag=3D, td=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:= 6136 #13 0xffffffff80464a55 in devfs_ioctl_f (fp=3D0xfffff80038bd00a0, com=3D322= 2821381, data=3D0xfffff800067b80a0, cred=3D, td=3D0xff= fff800628cc490) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #14 0xffffffff805f3c3d in kern_ioctl (td=3D0xfffff800628cc490, fd=3D, com=3D0) at file.h:311 #15 0xffffffff805f381c in sys_ioctl (td=3D0xfffff800628cc490, uap=3D0xfffff= e00955c6b80) at /usr/src/sys/kern/sys_generic.c:702 #16 0xffffffff8085c2db in amd64_syscall (td=3D0xfffff800628cc490, traced=3D= 0) at subr_syscall.c:133 #17 0xffffffff8083f90b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exce= ption.S:390 #18 0x00000008019fc3da in ?? () Previous frame inner to this frame (corrupt stack?) Thread 1188 (Thread 102480): #0 sched_switch (td=3D0xfffff80015a63920, newtd=3D, f= lags=3D) at /usr/src/sys/kern/sched_ule.c:1932 #1 0xffffffff805a23c1 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/src/= sys/kern/kern_synch.c:493 #2 0xffffffff805e4bca in sleepq_wait (wchan=3D0x0, pri=3D0) at /usr/src/sy= s/kern/subr_sleepqueue.c:631 #3 0xffffffff805a12b2 in _sx_xlock_hard (sx=3D0xfffff800062ed820, tid=3D18= 446735277979744544, opts=3D, file=3D0x0, line=3D0) at = /usr/src/sys/kern/kern_sx.c:676 #4 0xffffffff805a0add in _sx_xlock (sx=3D0x0, opts=3D0, file=3D, line=3D0) at sx.h:154 #5 0xffffffff8114a691 in spa_get_errlog_size (spa=3D0xfffff800062ed000) at= /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_errlog.c:142 #6 0xffffffff8113f549 in spa_get_stats (name=3D0xfffffe0005d6c000 "spacelo= op", config=3D0xfffffe0095f708e8, altroot=3D0xfffffe0005d6c430 "", buflen= =3D2048) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3287 #7 0xffffffff81189a45 in zfs_ioc_pool_stats (zc=3D0xfffffe0005d6c000) at /= usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1656 #8 0xffffffff81187290 in zfsdev_ioctl (dev=3D, zcmd= =3D, arg=3D, flag=3D, td=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:= 6136 #9 0xffffffff80464a55 in devfs_ioctl_f (fp=3D0xfffff8000631a320, com=3D322= 2821381, data=3D0xfffff800067b8d00, cred=3D, td=3D0xff= fff80015a63920) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #10 0xffffffff805f3c3d in kern_ioctl (td=3D0xfffff80015a63920, fd=3D, com=3D0) at file.h:311 #11 0xffffffff805f381c in sys_ioctl (td=3D0xfffff80015a63920, uap=3D0xfffff= e0095f70b80) at /usr/src/sys/kern/sys_generic.c:702 #12 0xffffffff8085c2db in amd64_syscall (td=3D0xfffff80015a63920, traced=3D= 0) at subr_syscall.c:133 #13 0xffffffff8083f90b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exce= ption.S:390 #14 0x00000008019fc3da in ?? () Previous frame inner to this frame (corrupt stack?) Fabian --Sig_/WvIg7bSSG29Dx92rRK+fj0F Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlQMeGQACgkQBYqIVf93VJ1EEQCgvi1CN0qc8FR7Xaqdh2efO47s d3AAoMH3hR4Kqy+Zy4nbgGx4KbSFEhE2 =fwhH -----END PGP SIGNATURE----- --Sig_/WvIg7bSSG29Dx92rRK+fj0F--