From owner-freebsd-stable@FreeBSD.ORG Tue Dec 13 22:15:04 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B3FD2106566B for ; Tue, 13 Dec 2011 22:15:04 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta12.westchester.pa.mail.comcast.net (qmta12.westchester.pa.mail.comcast.net [76.96.59.227]) by mx1.freebsd.org (Postfix) with ESMTP id 58C278FC13 for ; Tue, 13 Dec 2011 22:15:03 +0000 (UTC) Received: from omta10.westchester.pa.mail.comcast.net ([76.96.62.28]) by qmta12.westchester.pa.mail.comcast.net with comcast id 8m2H1i0040cZkys5CmF3AB; Tue, 13 Dec 2011 22:15:03 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta10.westchester.pa.mail.comcast.net with comcast id 8mF21i01a1t3BNj3WmF3EF; Tue, 13 Dec 2011 22:15:03 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6E6D2102C19; Tue, 13 Dec 2011 14:15:01 -0800 (PST) Date: Tue, 13 Dec 2011 14:15:01 -0800 From: Jeremy Chadwick To: Andrey Zonov Message-ID: <20111213221501.GA85563@icarus.home.lan> References: <4EE7BF77.5000504@zonov.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE7BF77.5000504@zonov.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org Subject: Re: directory listing hangs in "ufs" state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 22:15:04 -0000 On Wed, Dec 14, 2011 at 01:11:19AM +0400, Andrey Zonov wrote: > Hi, > > I've got STABLE-8 (r221983) with mongodb-1.8.1 installed on it. A > couple days ago I observed that listing of mongodb directory stuck > in a few minutes in "ufs" state. I've run it again with ktrace and > got following (kdump -R): > > 91324 ls 0.000003 CALL lstat(0x32c199c8,0x32c19950) > 91324 ls 0.000003 NAMI "base.1" > 91324 ls 21.357255 STRU struct stat {dev=116, ino=45125633, > mode=-rw------- , nlink=1, uid=922, gid=922, rdev=180226648, > atime=1323709877, stime=1323776461, ctime=1323776461, > birthtime=1314798592, size=134217728, blksize=16384, blocks=262304, > flags=0x0 } > 91324 ls 0.000014 RET lstat 0 > > kgdb backtrace of this process was looked like this: > > Thread 297 (Thread 100372): > #0 sched_switch (td=0xffffff0095c008c0, newtd=0xffffff000357b8c0, > flags=) at /usr/src/sys/kern/sched_ule.c:1866 > #1 0xffffffff80406696 in mi_switch (flags=260, newtd=0x0) at > /usr/src/sys/kern/kern_synch.c:449 > #2 0xffffffff8043c072 in sleepq_wait (wchan=0xffffff0103aaf7f8, > pri=80) at /usr/src/sys/kern/subr_sleepqueue.c:609 > #3 0xffffffff803e4a5a in __lockmgr_args (lk=0xffffff0103aaf7f8, > flags=2097408, ilk=0xffffff0103aaf820, wmesg=) at > /usr/src/sys/kern/kern_lock.c:220 > #5 0xffffffff8061239c in ffs_lock (ap=0xffffff84867fc550) at lockmgr.h:94 > #5 0xffffffff806d2462 in VOP_LOCK1_APV (vop=0xffffffff80921fe0, > a=0xffffff84867fc550) at vnode_if.c:1988 > #6 0xffffffff804a58b7 in _vn_lock (vp=0xffffff0103aaf760, > flags=2097152, file=0xffffffff80736e70 > "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859 > #7 0xffffffff80498bc0 in vget (vp=0xffffff0103aaf760, > flags=2097408, td=0xffffff0095c008c0) at > /usr/src/sys/kern/vfs_subr.c:2137 > #8 0xffffffff804845f4 in cache_lookup (dvp=0xffffff0095675b10, > vpp=0xffffff84867fc910, cnp=0xffffff84867fc938) at > /usr/src/sys/kern/vfs_cache.c:587 > #9 0xffffffff80484a30 in vfs_cache_lookup (ap=) at > /usr/src/sys/kern/vfs_cache.c:905 > #10 0xffffffff806d2e7c in VOP_LOOKUP_APV (vop=0xffffffff80922820, > a=0xffffff84867fc790) at vnode_if.c:123 > #11 0xffffffff8048bc80 in lookup (ndp=0xffffff84867fc8e0) at vnode_if.h:54 > #12 0xffffffff8048cf0e in namei (ndp=0xffffff84867fc8e0) at > /usr/src/sys/kern/vfs_lookup.c:269 > #13 0xffffffff8049c972 in kern_statat_vnhook (td=0xffffff0095c008c0, > flag=) at /usr/src/sys/kern/vfs_syscalls.c:2346 > #14 0xffffffff8049cbb5 in kern_statat (td=) at > /usr/src/sys/kern/vfs_syscalls.c:2327 > #15 0xffffffff8049cc7a in lstat (td=) at > /usr/src/sys/kern/vfs_syscalls.c:2390 > #16 0xffffffff8043e7dd in syscallenter (td=0xffffff0095c008c0, > sa=0xffffff84867fcbb0) at /usr/src/sys/kern/subr_trap.c:326 > #17 0xffffffff8066a5eb in syscall (frame=0xffffff84867fcc50) at > /usr/src/sys/amd64/amd64/trap.c:916 > #18 0xffffffff806517f2 in Xfast_syscall () at > /usr/src/sys/amd64/amd64/exception.S:384 > #19 0x000000003298f75c in ?? () > > The very first idea was to turn off name caching (set debug.vfscache > to 0), but it didn't help. The second idea was to reboot, but it > didn't help too. > > This directory locks fine. It has 10 files and 1 empty directory. > > Have you any ideas what is going on? or how to catch the problem? Assuming this isn't a file on the root filesystem, try booting the machine in single-user mode and using "fsck -f" on the filesystem in question. Can you verify there's no problems with the disk this file lives on as well (smartctl -a /dev/disk)? I'm doubting this is the problem, but thought I'd mention it. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |