Date: Thu, 25 Sep 2008 01:34:06 -0400 From: Jeff Wheelhouse <freebsd-hackers@wheelhouse.org> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-hackers@freebsd.org Subject: Re: panic: lockmgr on FreeBSD 7.0-RELEASE-p4 amd64 Message-ID: <57DCDBC7-8542-4082-8893-5B96DA92DA9A@wheelhouse.org> In-Reply-To: <200809241234.55075.jhb@freebsd.org> References: <D1048988-FAB7-4EDE-9EB8-8E01041C3F2A@wheelhouse.org> <BD22E0E4-D5B7-4C53-B3AF-461C0FA9A69C@wheelhouse.org> <200809241234.55075.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sep 24, 2008, at 12:34 PM, John Baldwin wrote:
> On Wednesday 24 September 2008 12:17:56 pm Jeff Wheelhouse wrote:
>> panic: lockmgr: thread 0xffffff0050858350, not exclusive lock holder
>> 0xffffff00074959f0 unlocking
>> cpuid = 0
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
>> panic() at panic+0x17a
>> _lockmgr() at _lockmgr+0x872
>> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
>> null_unlock() at null_unlock+0xff
>> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
>> nullfs_mount() at nullfs_mount+0x244
>> vfs_donmount() at vfs_donmount+0xe4d
>> nmount() at nmount+0xa5
>> syscall() at syscall+0x254
>> Xfast_syscall() at Xfast_syscall+0xab
>> --- syscall (378, FreeBSD ELF64, nmount), rip = 0x206845ac, rsp =
>> 0x7fffffffdfc8, rbp = 0x7fffffffdfd0 ---
>
> Can you use gdb or the like to get the souce file/line for the
> nullfs_mount+0x244 frame?
Got it again, this time with the full debug kernel, and I'm getting
the same weird results from gdb, so I'll go ahead and post it:
panic: lockmgr: thread 0xffffff0003e499f0, not exclusive lock holder
0xffffff000a5e16a0 unlocking
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x17a
_lockmgr() at _lockmgr+0x872
VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
null_unlock() at null_unlock+0xff
VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
nullfs_mount() at nullfs_mount+0x244
vfs_donmount() at vfs_donmount+0xe4d
nmount() at nmount+0xa5
syscall() at syscall+0x254
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (378, FreeBSD ELF64, nmount), rip = 0x206845ac, rsp =
0x7fffffffe1c8, rbp = 0x7fffffffe1d0 ---
$ gdb /boot/kernel/nullfs.ko
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...
(gdb) l *nullfs_mount+0x244
0x9c4 is in nullfs_mount (namei.h:163).
158 struct thread *td)
159 {
160 ndp->ni_cnd.cn_nameiop = op;
161 ndp->ni_cnd.cn_flags = flags;
162 ndp->ni_segflg = segflg;
163 ndp->ni_dirp = namep;
164 ndp->ni_cnd.cn_thread = td;
165 }
166
167 #define NDF_NO_DVP_RELE 0x00000001
(gdb)
(That's NDINIT(), but line 163 doesn't look like it belongs in the
middle of a call stack. There's a VOP_UNLOCK a few lines above
NDINIT() in mount_nullfs(), and another one some ways farther on in
the function.)
The good news is we took this particular machine out of production and
came up with a synthetic test based on our in-house code that can
probably reliably reproduce this within a few minutes. As you might
expect, the test involves hammering the same nullfs mount point with
mounts and umounts from multiple processes without any external
synchronization.
Thanks,
Jeff
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?57DCDBC7-8542-4082-8893-5B96DA92DA9A>