Date: Tue, 1 Feb 2011 09:46:16 -0500 From: John Baldwin <jhb@freebsd.org> To: freebsd-fs@freebsd.org Subject: Re: LOR in dirhash Message-ID: <201102010946.16466.jhb@freebsd.org> In-Reply-To: <4D47D847.8080308@bytecamp.net> References: <4D47D847.8080308@bytecamp.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tuesday, February 01, 2011 4:54:15 am Robert Schulze wrote: > We are currently having continuous problems with 8-STABLE. > Since we upgraded, we have "double faults" every three days or so,=20 > yesterday we built a debugging kernel to identify the problem. >=20 > Today I saw a LOR in /var/log/messages, the whole output: >=20 > 8<---------------------------------------------------------- >=20 > lock order reversal: > 1st 0xffffff81ef08ce98 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:26= 36 > 2nd 0xffffff00157b4600 dirhash (dirhash) @=20 > /usr/src/sys/ufs/ufs/ufs_dirhash.c:285 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > _witness_debugger() at _witness_debugger+0x49 > witness_checkorder() at witness_checkorder+0x7d3 > _sx_xlock() at _sx_xlock+0x4a > ufsdirhash_acquire() at ufsdirhash_acquire+0x3a > ufsdirhash_add() at > ufsdirhash_add+0x19 > ufs_diren > ter() at > ufs_direnter+0x876 > ufs_makeinode() at > ufs_makeinode+0x239 > VOP_CREATE_APV() at > VOP_CREATE_APV+0xb6 > vn_open_cred() at vn_open_cred+0x415 > kern_openat() at > kern_openat+0x165 > syscallenter() at syscallenter+0xe5 > syscall() at > syscall+0x55 > Xfast_syscall() at > Xfast_syscall+0xe2 > --- syscall (5, FreeBSD ELF64 > , open) > , rip =3D > 0x8009 > a4a7c, > rsp =3D 0x > 7ffffff > e918, r > bp =3D 0x1 --- >=20 > 8<---------------------------------------------------------- >=20 > maybe this is a seriuos one? =46rom the source code in ufs_dirhash.c: /* * Locking: * * The relationship between inode and dirhash is protected either by an * exclusive vnode lock or the vnode interlock where a shared vnode lock * may be used. The dirhash_mtx is acquired after the dirhash lock. To * handle teardown races, code wishing to lock the dirhash for an inode * when using a shared vnode lock must obtain a private reference on the * dirhash while holding the vnode interlock. They can drop it once they * have obtained the dirhash lock and verified that the dirhash wasn't * recycled while they waited for the dirhash lock. * * ufsdirhash_build() acquires a shared lock on the dirhash when it is * successful. This lock is released after a call to ufsdirhash_lookup(). * * Functions requiring exclusive access use ufsdirhash_acquire() which may * free a dirhash structure that was recycled by ufsdirhash_recycle(). * * The dirhash lock may be held across io operations. * * WITNESS reports a lock order reversal between the "bufwait" lock * and the "dirhash" lock. However, this specific reversal will not * cause a deadlock. To get a deadlock, one would have to lock a * buffer followed by the dirhash while a second thread locked a * buffer while holding the dirhash lock. The second order can happen * under a shared or exclusive vnode lock for the associated directory * in lookup(). The first order, however, can only happen under an * exclusive vnode lock (e.g. unlink(), rename(), etc.). Thus, for * a thread to be doing a "bufwait" -> "dirhash" order, it has to hold * an exclusive vnode lock. That exclusive vnode lock will prevent * any other threads from doing a "dirhash" -> "bufwait" order. */ =2D-=20 John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201102010946.16466.jhb>