Date: Tue, 13 Dec 2011 14:15:01 -0800 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Andrey Zonov <andrey@zonov.org> Cc: freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state Message-ID: <20111213221501.GA85563@icarus.home.lan> In-Reply-To: <4EE7BF77.5000504@zonov.org> References: <4EE7BF77.5000504@zonov.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote:
> Hi,
>
> I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it. A
> couple days ago I observed that listing of mongodb directory stuck
> in a few minutes in "ufs" state. I've run it again with ktrace and
> got following (kdump -R):
>
> 91324 ls 0.000003 CALL lstat(0x32c199c8,0x32c19950)
> 91324 ls 0.000003 NAMI "base.1"
> 91324 ls 21.357255 STRU struct stat {dev=116, ino=45125633,
> mode=-rw------- , nlink=1, uid=922, gid=922, rdev=180226648,
> atime=1323709877, stime=1323776461, ctime=1323776461,
> birthtime=1314798592, size=134217728, blksize=16384, blocks=262304,
> flags=0x0 }
> 91324 ls 0.000014 RET lstat 0
>
> kgdb backtrace of this process was looked like this:
>
> Thread 297 (Thread 100372):
> #0 sched_switch (td=0xffffff0095c008c0, newtd=0xffffff000357b8c0,
> flags=) at /usr/src/sys/kern/sched_ule.c:1866
> #1 0xffffffff80406696 in mi_switch (flags=260, newtd=0x0) at
> /usr/src/sys/kern/kern_synch.c:449
> #2 0xffffffff8043c072 in sleepq_wait (wchan=0xffffff0103aaf7f8,
> pri=80) at /usr/src/sys/kern/subr_sleepqueue.c:609
> #3 0xffffffff803e4a5a in __lockmgr_args (lk=0xffffff0103aaf7f8,
> flags=2097408, ilk=0xffffff0103aaf820, wmesg=) at
> /usr/src/sys/kern/kern_lock.c:220
> #5 0xffffffff8061239c in ffs_lock (ap=0xffffff84867fc550) at lockmgr.h:94
> #5 0xffffffff806d2462 in VOP_LOCK1_APV (vop=0xffffffff80921fe0,
> a=0xffffff84867fc550) at vnode_if.c:1988
> #6 0xffffffff804a58b7 in _vn_lock (vp=0xffffff0103aaf760,
> flags=2097152, file=0xffffffff80736e70
> "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859
> #7 0xffffffff80498bc0 in vget (vp=0xffffff0103aaf760,
> flags=2097408, td=0xffffff0095c008c0) at
> /usr/src/sys/kern/vfs_subr.c:2137
> #8 0xffffffff804845f4 in cache_lookup (dvp=0xffffff0095675b10,
> vpp=0xffffff84867fc910, cnp=0xffffff84867fc938) at
> /usr/src/sys/kern/vfs_cache.c:587
> #9 0xffffffff80484a30 in vfs_cache_lookup (ap=) at
> /usr/src/sys/kern/vfs_cache.c:905
> #10 0xffffffff806d2e7c in VOP_LOOKUP_APV (vop=0xffffffff80922820,
> a=0xffffff84867fc790) at vnode_if.c:123
> #11 0xffffffff8048bc80 in lookup (ndp=0xffffff84867fc8e0) at vnode_if.h:54
> #12 0xffffffff8048cf0e in namei (ndp=0xffffff84867fc8e0) at
> /usr/src/sys/kern/vfs_lookup.c:269
> #13 0xffffffff8049c972 in kern_statat_vnhook (td=0xffffff0095c008c0,
> flag=) at /usr/src/sys/kern/vfs_syscalls.c:2346
> #14 0xffffffff8049cbb5 in kern_statat (td=) at
> /usr/src/sys/kern/vfs_syscalls.c:2327
> #15 0xffffffff8049cc7a in lstat (td=) at
> /usr/src/sys/kern/vfs_syscalls.c:2390
> #16 0xffffffff8043e7dd in syscallenter (td=0xffffff0095c008c0,
> sa=0xffffff84867fcbb0) at /usr/src/sys/kern/subr_trap.c:326
> #17 0xffffffff8066a5eb in syscall (frame=0xffffff84867fcc50) at
> /usr/src/sys/amd64/amd64/trap.c:916
> #18 0xffffffff806517f2 in Xfast_syscall () at
> /usr/src/sys/amd64/amd64/exception.S:384
> #19 0x000000003298f75c in ?? ()
>
> The very first idea was to turn off name caching (set debug.vfscache
> to 0), but it didn't help. The second idea was to reboot, but it
> didn't help too.
>
> This directory locks fine. It has 10 files and 1 empty directory.
>
> Have you any ideas what is going on? or how to catch the problem?
Assuming this isn't a file on the root filesystem, try booting the
machine in single-user mode and using "fsck -f" on the filesystem in
question.
Can you verify there's no problems with the disk this file lives on as
well (smartctl -a /dev/disk)? I'm doubting this is the problem, but
thought I'd mention it.
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, US |
| Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111213221501.GA85563>