Date: Thu, 22 May 2003 02:30:46 +0900 (JST) From: Tod McQuillin <devin@spamcop.net> To: Mike Harding <mvh@ix.netcom.com> Cc: stable@freebsd.org Subject: Re: system slowdown - vnode related Message-ID: <20030522022137.D551@borosilicate.pun-pun.prv> In-Reply-To: <20030521171941.364325314@netcom1.netcom.com> References: <20030521171941.364325314@netcom1.netcom.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 21 May 2003, Mike Harding wrote:
> I woke up to a frozen box this morning - it froze up a few more times
> before I got a handle on it.
>
> Basically, the box runs idle but refuses to do disk IO, or does it
> -very- slowly.
>
> Top shows processes stuck in 'ffsvget', 'inode', and 'vlruwk' state.
[...]
> I can reliably get the box in to this state by doing 'find /'. I do
> have a lot of files on the disk, and a things like squid and postgres
> that do a lot of file i/o, but I don't recall this happening before
> this week. I don't find anything in 'tuning' about bumping up vnodes,
> but I do see sporadic reports on a google group search - searching for
> 'ffsvgt'.
>
> Anybody run into this before?
I can reproduce this at will with rsync. Here's how:
1) check out ports from the ports cvs repo using 'cvs co ports' (you can
use an anoncvs server or your own local ports cvs repo from cvsup)
2) Since all the CVS/Root files are the same, change them all into hard
links to the same file with a script like this:
#!/usr/bin/perl
#
# First arg: file to link to
#
# remaining args: files to check if their contents is the same as first
# file. If they are the same, they will be removed and replaced with a
# hard link to the first file.
my $master = shift @ARGV;
print "Master file: $master\n";
local $/ = undef;
open(M, $master) || die "$master: $!\n";
my $master_content = <M>;
close M;
while (my $file = shift @ARGV) {
if ($file eq $master) {
print "Skipping original $master\n";
next;
}
open (F, $file) || die "$file: $!\n";
my $content = <F>;
close F;
if ($content eq $master_content) {
print "replacing $file\n";
unlink $file or die "$file: $!\n";
link $master, $file or die "link $master, $file: $!\n";
}
}
then:
cd /usr/ports; find . -name Root | xargs /tmp/link-if-same.pl ./CVS/Root
This will produce thousands of hard links to the same file, like:
/usr/ports% ls -l CVS/Root
-rw-r--r-- 13613 root wheel 18 May 19 15:03 CVS/Root
3) rsync your ports tree to another box with
rsync -avHS source::ports/ /usr/ports/
(set up /usr/local/etc/rsyncd.conf and /etc/inetd.conf as appropriate)
4) watch the destination box hang with rsync in vlruwk wchan.
Yes I know this is convoluted, but it reliably reproduces the same problem
for me. In my case the fix was to use --exclude=CVS/ in the rsync flags
-- I'm sure that doesn't help you.
No, I don't have a fix, but maybe this recipe will help someone else debug
it.
--
Tod McQuillin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030522022137.D551>