From owner-freebsd-hackers@FreeBSD.ORG Wed Sep 15 19:09:57 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6649D1065696 for ; Wed, 15 Sep 2010 19:09:54 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id C3C8B8FC1A for ; Wed, 15 Sep 2010 19:09:53 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 4AA5A46C20; Wed, 15 Sep 2010 15:09:53 -0400 (EDT) Received: from jhbbsd.localnet (smtp.hudson-trading.com [209.249.190.9]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4FECA8A04F; Wed, 15 Sep 2010 15:09:52 -0400 (EDT) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Wed, 15 Sep 2010 15:09:49 -0400 User-Agent: KMail/1.13.5 (FreeBSD/7.3-CBSD-20100819; KDE/4.4.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201009151509.49728.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Wed, 15 Sep 2010 15:09:52 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: cronfy Subject: Re: is vfs.lookup_shared unsafe in 7.3? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Sep 2010 19:09:57 -0000 On Monday, September 13, 2010 4:57:15 pm cronfy wrote: > Hello, > > Trying to overtake high server load (sudden peaks of 15%us/85%sy, LA > > 40, very slow lstat() at these moments, looks like some kind of lock > contention) I enabled vfs.lookup_shared=1 on two servers today. One is > FreeBSD-7.3 kernel csup'ed and built Sep 9 2010 and other is > FreeBSD-7.3 csup'ed and built Jul 16 2010. > > The server with more fresh kernel is running nice and does not show > high load anymore. But on the second server it did not help. More, > after a few hours of work with vfs.lookup_shared=1 I noticed processes > stucked in "ufs" state. I tried to kill them with no luck. Disabling > vfs.lookup_shared freezed the whole system. > > So, is vfs.lookup_shared=1 unsafe in 7.3? Did it become more stable > between 16 Jul and 9 Sep (is it the reason why first system is still > running?), or should I expect that it will freeze in a near time too? > > Thanks in advance! No, 7.3 has a bug that can cause these hangs that is probably made worse by vfs.lookup_shared=1, but can occur even if it is disabled. You want these fixes applied (in order, one of them reverts part of another): Author: jhb Date: Fri Jul 16 20:23:24 2010 New Revision: 210173 URL: http://svn.freebsd.org/changeset/base/210173 Log: When the MNTK_EXTENDED_SHARED mount option was added, some filesystems were changed to defer the setting of VN_LOCK_ASHARE() (which clears LK_NOSHARE in the vnode lock's flags) until after they had determined if the vnode was a FIFO. This occurs after the vnode has been inserted into a VFS hash or some similar table, so it is possible for another thread to find this vnode via vget() on an i-node number and block on the vnode lock. If the lockmgr interlock (vnode interlock for vnode locks) is not held when clearing the LK_NOSHARE flag, then the lk_flags field can be clobbered. As a result the thread blocked on the vnode lock may never get woken up. Fix this by holding the vnode interlock while modifying the lock flags in this case. The softupdates code also toggles LK_NOSHARE in one function to close a race with snapshots. Fix this code to grab the interlock while fiddling with lk_flags. Modified: stable/7/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c stable/7/sys/fs/cd9660/cd9660_vfsops.c stable/7/sys/fs/udf/udf_vfsops.c stable/7/sys/ufs/ffs/ffs_softdep.c stable/7/sys/ufs/ffs/ffs_vfsops.c Author: jhb Date: Fri Aug 20 20:33:13 2010 New Revision: 211532 URL: http://svn.freebsd.org/changeset/base/211532 Log: MFC: Use VN_LOCK_AREC() and VN_LOCK_ASHARE() rather than manipulating lockmgr lock flags directly. Modified: stable/7/sys/fs/nwfs/nwfs_node.c stable/7/sys/fs/pseudofs/pseudofs_vncache.c stable/7/sys/fs/smbfs/smbfs_node.c stable/7/sys/gnu/fs/xfs/FreeBSD/xfs_freebsd_iget.c stable/7/sys/kern/vfs_lookup.c Author: jhb Date: Fri Aug 20 20:58:57 2010 New Revision: 211533 URL: http://svn.freebsd.org/changeset/base/211533 Log: Revert 210173 as it did not properly fix the bug. It assumed that the VI_LOCK() for a given vnode was used as the internal interlock for that vnode's v_lock lockmgr lock. This is not the case. Instead, add dedicated routines to toggle the LK_NOSHARE and LK_CANRECURSE flags. These routines lock the lockmgr lock's internal interlock to synchronize the updates to the flags member with other threads attempting to acquire the lock. The VN_LOCK_A*() macros now invoke these routines, and the softupdates code uses these routines to temporarly enable recursion on buffer locks. Reviewed by: kib Modified: stable/7/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c stable/7/sys/fs/cd9660/cd9660_vfsops.c stable/7/sys/fs/udf/udf_vfsops.c stable/7/sys/kern/kern_lock.c stable/7/sys/sys/lockmgr.h stable/7/sys/sys/vnode.h stable/7/sys/ufs/ffs/ffs_softdep.c stable/7/sys/ufs/ffs/ffs_vfsops.c -- John Baldwin