From owner-freebsd-stable@FreeBSD.ORG Wed May 21 10:30:51 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A8F3D37B401 for ; Wed, 21 May 2003 10:30:51 -0700 (PDT) Received: from mail.distalzou.net (203.141.139.231.user.ad.il24.net [203.141.139.231]) by mx1.FreeBSD.org (Postfix) with ESMTP id AFAC343F3F for ; Wed, 21 May 2003 10:30:48 -0700 (PDT) (envelope-from devin@spamcop.net) Received: from borosilicate.pun-pun.prv ([192.168.7.29]) by mail.distalzou.net with esmtp (TLSv1:AES256-SHA:256) (Exim 4.20) id 19IXQM-00040o-I0; Thu, 22 May 2003 02:30:46 +0900 Date: Thu, 22 May 2003 02:30:46 +0900 (JST) From: Tod McQuillin X-X-Sender: devin@borosilicate.pun-pun.prv To: Mike Harding In-Reply-To: <20030521171941.364325314@netcom1.netcom.com> Message-ID: <20030522022137.D551@borosilicate.pun-pun.prv> References: <20030521171941.364325314@netcom1.netcom.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: stable@freebsd.org Subject: Re: system slowdown - vnode related X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 May 2003 17:30:51 -0000 On Wed, 21 May 2003, Mike Harding wrote: > I woke up to a frozen box this morning - it froze up a few more times > before I got a handle on it. > > Basically, the box runs idle but refuses to do disk IO, or does it > -very- slowly. > > Top shows processes stuck in 'ffsvget', 'inode', and 'vlruwk' state. [...] > I can reliably get the box in to this state by doing 'find /'. I do > have a lot of files on the disk, and a things like squid and postgres > that do a lot of file i/o, but I don't recall this happening before > this week. I don't find anything in 'tuning' about bumping up vnodes, > but I do see sporadic reports on a google group search - searching for > 'ffsvgt'. > > Anybody run into this before? I can reproduce this at will with rsync. Here's how: 1) check out ports from the ports cvs repo using 'cvs co ports' (you can use an anoncvs server or your own local ports cvs repo from cvsup) 2) Since all the CVS/Root files are the same, change them all into hard links to the same file with a script like this: #!/usr/bin/perl # # First arg: file to link to # # remaining args: files to check if their contents is the same as first # file. If they are the same, they will be removed and replaced with a # hard link to the first file. my $master = shift @ARGV; print "Master file: $master\n"; local $/ = undef; open(M, $master) || die "$master: $!\n"; my $master_content = ; close M; while (my $file = shift @ARGV) { if ($file eq $master) { print "Skipping original $master\n"; next; } open (F, $file) || die "$file: $!\n"; my $content = ; close F; if ($content eq $master_content) { print "replacing $file\n"; unlink $file or die "$file: $!\n"; link $master, $file or die "link $master, $file: $!\n"; } } then: cd /usr/ports; find . -name Root | xargs /tmp/link-if-same.pl ./CVS/Root This will produce thousands of hard links to the same file, like: /usr/ports% ls -l CVS/Root -rw-r--r-- 13613 root wheel 18 May 19 15:03 CVS/Root 3) rsync your ports tree to another box with rsync -avHS source::ports/ /usr/ports/ (set up /usr/local/etc/rsyncd.conf and /etc/inetd.conf as appropriate) 4) watch the destination box hang with rsync in vlruwk wchan. Yes I know this is convoluted, but it reliably reproduces the same problem for me. In my case the fix was to use --exclude=CVS/ in the rsync flags -- I'm sure that doesn't help you. No, I don't have a fix, but maybe this recipe will help someone else debug it. -- Tod McQuillin