Date: Sun, 7 Sep 2014 17:23:16 +0200 From: Fabian Keil <freebsd-listen@fabiankeil.de> To: freebsd-current@freebsd.org Subject: Re: ZFS-related panic: "possible" spa->spa_errlog_lock deadlock Message-ID: <4fa875ba.3cc970d7@fabiankeil.de> In-Reply-To: <540C66AC.8070809@delphij.net> References: <492dbacb.5942cc9b@fabiankeil.de> <540C66AC.8070809@delphij.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--Sig_/WvIg7bSSG29Dx92rRK+fj0F Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Xin Li <delphij@delphij.net> wrote: > On 9/7/14 9:02 PM, Fabian Keil wrote: > > Using a kernel built from FreeBSD 11.0-CURRENT r271182 I got the > > following panic yesterday: > >=20 > > [...] Unread portion of the kernel message buffer: [6880] panic: > > deadlkres: possible deadlock detected for 0xfffff80015289490, > > blocked for 1800503 ticks >=20 > Any chance to get all backtraces (e.g. thread apply all bt full 16)? > I think a different thread that held the lock have been blocked, > probably related to your disconnected vdev. Output of "thread apply all bt full 16" is available at: http://www.fabiankeil.de/tmp/freebsd/kgdb-output-spa_errlog_lock-deadlock.t= xt A lot of the backtraces prematurely end with "Cannot access memory at addre= ss", therefore I also added "thread apply all bt" output. Apparently there are at least two additional threads blocking below spa_get= _stats(): Thread 1182 (Thread 101989): #0 sched_switch (td=3D0xfffff800628cc490, newtd=3D<value optimized out>, f= lags=3D<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1932 #1 0xffffffff805a23c1 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/src/= sys/kern/kern_synch.c:493 #2 0xffffffff805e4bca in sleepq_wait (wchan=3D0x0, pri=3D0) at /usr/src/sy= s/kern/subr_sleepqueue.c:631 #3 0xffffffff80539f10 in _cv_wait (cvp=3D0xfffff80025534a50, lock=3D0xffff= f80025534a30) at /usr/src/sys/kern/kern_condvar.c:139 #4 0xffffffff811721db in zio_wait (zio=3D<value optimized out>) at /usr/sr= c/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1442 #5 0xffffffff81102111 in dbuf_read (db=3D<value optimized out>, zio=3D<val= ue optimized out>, flags=3D<value optimized out>) at /usr/src/sys/cddl/cont= rib/opensolaris/uts/common/fs/zfs/dbuf.c:649 #6 0xffffffff81108e6d in dmu_buf_hold (os=3D<value optimized out>, object= =3D<value optimized out>, offset=3D<value optimized out>, tag=3D0x0, dbp=3D= 0xfffffe00955c6648, flags=3D<value optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:172 #7 0xffffffff81163986 in zap_lockdir (os=3D0xfffff8002b7ab000, obj=3D92, t= x=3D0x0, lti=3DRW_READER, fatreader=3D1, adding=3D0, zapp=3D<value optimize= d out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:= 467 #8 0xffffffff811644ad in zap_count (os=3D0x0, zapobj=3D0, count=3D0xfffffe= 00955c66d8) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_= micro.c:712 #9 0xffffffff8114a6dc in spa_get_errlog_size (spa=3D0xfffff800062ed000) at= /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_errlog.c:149 ---Type <return> to continue, or q <return> to quit--- #10 0xffffffff8113f549 in spa_get_stats (name=3D0xfffffe0044cac000 "spacelo= op", config=3D0xfffffe00955c68e8, altroot=3D0xfffffe0044cac430 "", buflen= =3D2048) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3287 #11 0xffffffff81189a45 in zfs_ioc_pool_stats (zc=3D0xfffffe0044cac000) at /= usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1656 #12 0xffffffff81187290 in zfsdev_ioctl (dev=3D<value optimized out>, zcmd= =3D<value optimized out>, arg=3D<value optimized out>, flag=3D<value optimi= zed out>, td=3D<value optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:= 6136 #13 0xffffffff80464a55 in devfs_ioctl_f (fp=3D0xfffff80038bd00a0, com=3D322= 2821381, data=3D0xfffff800067b80a0, cred=3D<value optimized out>, td=3D0xff= fff800628cc490) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #14 0xffffffff805f3c3d in kern_ioctl (td=3D0xfffff800628cc490, fd=3D<value = optimized out>, com=3D0) at file.h:311 #15 0xffffffff805f381c in sys_ioctl (td=3D0xfffff800628cc490, uap=3D0xfffff= e00955c6b80) at /usr/src/sys/kern/sys_generic.c:702 #16 0xffffffff8085c2db in amd64_syscall (td=3D0xfffff800628cc490, traced=3D= 0) at subr_syscall.c:133 #17 0xffffffff8083f90b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exce= ption.S:390 #18 0x00000008019fc3da in ?? () Previous frame inner to this frame (corrupt stack?) Thread 1188 (Thread 102480): #0 sched_switch (td=3D0xfffff80015a63920, newtd=3D<value optimized out>, f= lags=3D<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1932 #1 0xffffffff805a23c1 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/src/= sys/kern/kern_synch.c:493 #2 0xffffffff805e4bca in sleepq_wait (wchan=3D0x0, pri=3D0) at /usr/src/sy= s/kern/subr_sleepqueue.c:631 #3 0xffffffff805a12b2 in _sx_xlock_hard (sx=3D0xfffff800062ed820, tid=3D18= 446735277979744544, opts=3D<value optimized out>, file=3D0x0, line=3D0) at = /usr/src/sys/kern/kern_sx.c:676 #4 0xffffffff805a0add in _sx_xlock (sx=3D0x0, opts=3D0, file=3D<value opti= mized out>, line=3D0) at sx.h:154 #5 0xffffffff8114a691 in spa_get_errlog_size (spa=3D0xfffff800062ed000) at= /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa_errlog.c:142 #6 0xffffffff8113f549 in spa_get_stats (name=3D0xfffffe0005d6c000 "spacelo= op", config=3D0xfffffe0095f708e8, altroot=3D0xfffffe0005d6c430 "", buflen= =3D2048) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3287 #7 0xffffffff81189a45 in zfs_ioc_pool_stats (zc=3D0xfffffe0005d6c000) at /= usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1656 #8 0xffffffff81187290 in zfsdev_ioctl (dev=3D<value optimized out>, zcmd= =3D<value optimized out>, arg=3D<value optimized out>, flag=3D<value optimi= zed out>, td=3D<value optimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:= 6136 #9 0xffffffff80464a55 in devfs_ioctl_f (fp=3D0xfffff8000631a320, com=3D322= 2821381, data=3D0xfffff800067b8d00, cred=3D<value optimized out>, td=3D0xff= fff80015a63920) at /usr/src/sys/fs/devfs/devfs_vnops.c:757 #10 0xffffffff805f3c3d in kern_ioctl (td=3D0xfffff80015a63920, fd=3D<value = optimized out>, com=3D0) at file.h:311 #11 0xffffffff805f381c in sys_ioctl (td=3D0xfffff80015a63920, uap=3D0xfffff= e0095f70b80) at /usr/src/sys/kern/sys_generic.c:702 #12 0xffffffff8085c2db in amd64_syscall (td=3D0xfffff80015a63920, traced=3D= 0) at subr_syscall.c:133 #13 0xffffffff8083f90b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exce= ption.S:390 #14 0x00000008019fc3da in ?? () Previous frame inner to this frame (corrupt stack?) Fabian --Sig_/WvIg7bSSG29Dx92rRK+fj0F Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlQMeGQACgkQBYqIVf93VJ1EEQCgvi1CN0qc8FR7Xaqdh2efO47s d3AAoMH3hR4Kqy+Zy4nbgGx4KbSFEhE2 =fwhH -----END PGP SIGNATURE----- --Sig_/WvIg7bSSG29Dx92rRK+fj0F--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4fa875ba.3cc970d7>