Date: Sat, 1 Aug 2009 19:14:10 +0200 From: Attilio Rao <attilio@freebsd.org> To: Linda Messerschmidt <linda.messerschmidt@gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: Where have all the vnodes gone? Message-ID: <3bbf2fe10908011014r2fda9245xc7c5f71fcc544d10@mail.gmail.com> In-Reply-To: <237c27100908010841g5abd3183w479121b2ba7b0c91@mail.gmail.com> References: <237c27100908010841g5abd3183w479121b2ba7b0c91@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
2009/8/1 Linda Messerschmidt <linda.messerschmidt@gmail.com>: > (Reposted from freebsd-questions due to no replies.) > > With the last few releases, I've noticed a distinct trend toward > disappearing vnodes on one of the machines I look after. > > This machine isn't doing a whole lot. It runs a couple of small web > sites, and once an hour it rsync's some files from one NFS mount to > another, but the rsync doesn't stay running; it restarts every hour > and runs for 10-15 minutes. > > I set it to log the number of vnodes every ten minutes and this is what I got: > > 00:47:59 vfs.numvnodes: 39337 > 00:57:59 vfs.numvnodes: 40568 > 01:07:59 vfs.numvnodes: 44554 > 01:17:59 vfs.numvnodes: 52141 > 01:27:59 vfs.numvnodes: 55713 > 01:37:59 vfs.numvnodes: 58643 > 01:47:59 vfs.numvnodes: 60792 > 01:57:59 vfs.numvnodes: 67130 > 02:07:59 vfs.numvnodes: 76035 > 02:17:59 vfs.numvnodes: 84349 > 02:27:59 vfs.numvnodes: 92187 > 02:37:59 vfs.numvnodes: 98114 > 02:47:59 vfs.numvnodes: 116854 > 02:57:59 vfs.numvnodes: 124842 > 03:07:59 vfs.numvnodes: 164173 > 03:17:59 vfs.numvnodes: 172257 > 03:27:59 vfs.numvnodes: 178388 > 03:37:59 vfs.numvnodes: 183066 > 03:47:59 vfs.numvnodes: 190092 > 03:57:59 vfs.numvnodes: 198322 > 04:07:59 vfs.numvnodes: 204598 > 04:17:59 vfs.numvnodes: 208311 > 04:27:59 vfs.numvnodes: 214207 > 04:37:59 vfs.numvnodes: 221028 > 04:47:59 vfs.numvnodes: 227792 > 04:57:59 vfs.numvnodes: 233214 > 05:07:59 vfs.numvnodes: 240112 > 05:17:59 vfs.numvnodes: 247572 > 05:27:59 vfs.numvnodes: 256090 > 05:37:59 vfs.numvnodes: 262720 > 05:47:59 vfs.numvnodes: 269755 > 05:57:59 vfs.numvnodes: 274986 > 06:07:59 vfs.numvnodes: 279879 > 06:17:59 vfs.numvnodes: 287039 > 06:27:59 vfs.numvnodes: 291984 > 06:37:59 vfs.numvnodes: 294267 > 06:47:59 vfs.numvnodes: 296658 > 06:57:59 vfs.numvnodes: 299086 > 07:07:59 vfs.numvnodes: 301825 > 07:17:59 vfs.numvnodes: 309060 > 07:27:59 vfs.numvnodes: 312955 > 07:37:59 vfs.numvnodes: 317400 > 07:47:59 vfs.numvnodes: 320047 > > At that point the machine crashed with: > > panic: kmem_malloc(16384): kmem_map too small: 334745600 total allocated > > If I don't tune kern.maxvnodes up to the point where it panics, then > eventually it runs out of vnodes and all sorts of stuff gets stuck in > vlruwk. > > The machine in question is running 7.2-RELEASE-p3, but I already > upgraded it from 7.1 trying to get this to go away, so it's a problem > that's been around for awhile. > > My guess is that they're leaking in the kernel somewhere because of > the rsync, because there's just not much else going on, but unless I > can figure out how many vnodes are being used on a per-process basis, > I can't make any headway on proving or disproving that. I do know > that according to fstat, there are only 1000-1500 descriptors open at > any given time, and kern.openfiles ranges 250-500. So I'm just > mystified what the other 300000+ could be. > > Is there any way to figure out where all these vnodes are going? It seems you can reproduce it easilly. Can you please provide KTR traces of KTR_VFS? Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3bbf2fe10908011014r2fda9245xc7c5f71fcc544d10>