Date: 27 May 2003 11:27:30 -0700 From: Mike Harding <mvh@ix.netcom.com> To: Matthew Dillon <dillon@apollo.backplane.com> Cc: stable@freebsd.org Subject: Re: system slowdown - vnode related Message-ID: <1054060050.640.35.camel@netcom1.netcom.com> In-Reply-To: <200305271809.h4RI9ssD066960@apollo.backplane.com> References: <20030521171941.364325314@netcom1.netcom.com> <20030524190051.R598@hub.org><20030526123617.D56519@hub.org> <1053964809.7831.6.camel@netcom1.netcom.com> <20030526130556.G56519@hub.org> <200305271809.h4RI9ssD066960@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I'll try this if I can tickle the bug again.
I may have just run out of freevnodes - I only have about 1-2000 free
right now. I was just surprised because I have never seen a reference
to tuning this sysctl.
- Mike H.
On Tue, 2003-05-27 at 11:09, Matthew Dillon wrote:
> I'm a little confused. What state is the vnlru kernel thread in? It
> sounds like vnlru must be stuck.
>
> Note that you can gdb the live kernel and get a stack backtrace of any
> stuck process.
>
> gdb -k /kernel.debug /dev/mem (or whatever)
> proc N (e.g. vnlru's pid)
> back
>
> All the processes stuck in 'inode' are likely associated with the
> problem, but if that is what is causing vnlru to be stuck I would expect
> vnlru itself to be stuck in 'inode'.
>
> unionfs is probably responsible. I would not be surprised at all if
> unionfs is causing a deadlock somewhere which is creating a chain of
> processes stuck in 'inode' which is in turn causing vnlru to get stuck.
>
> -Matt
> Matthew Dillon
> <dillon@backplane.com>
>
> :
> :On Mon, 26 May 2003, Mike Harding wrote:
> :
> :> Er - are any changes made to RELENG_4_8 that aren't made to RELENG_4? I
> :> thought it was the other way around - that 4_8 only got _some_ of the
> :> changes to RELENG_4...
> :
> :Ack, my fault ... sorry, wasn't thinking :( RELENG_4 is correct ... I
> :should have confirmed my settings before blathering on ...
> :
> :One of the scripts I used extensively while debugging this ... a quite
> :simple one .. was:
> :
> :#!/bin/tcsh
> :while ( 1 )
> : echo `sysctl debug.numvnodes` - `sysctl debug.freevnodes` - `sysctl debug.vnlru_nowhere` - `ps auxl | grep vnlru | grep -v grep | awk '{print $20}'`
> : sleep 10
> :end
> :
> :which outputs this:
> :
> :debug.numvnodes: 463421 - debug.freevnodes: 220349 - debug.vnlru_nowhere: 3 - vlruwt
> :
> :I have my maxvnodes set to 512k right now ... now, when the server "hung",
> :the output would look something like (this would be with 'default' vnodes):
> :
> :debug.numvnodes: 199252 - debug.freevnodes: 23 - debug.vnlru_nowhere: 12 - vlrup
> :
> :with the critical bit being the vlruwt -> vlrup change ...
> :
> :with unionfs, you are using two vnodes per file, instead of one in
> :non-union mode, which is why I went to 512k vs the default of ~256k vnodes
> :... it doesn't *fix* the problem, it only reduces its occurance ...
> :_______________________________________________
> :freebsd-stable@freebsd.org mailing list
> :http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> :To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
> :
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1054060050.640.35.camel>