From owner-freebsd-hackers@FreeBSD.ORG Sat Aug 1 17:37:24 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 443C6106566C for ; Sat, 1 Aug 2009 17:37:24 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-fx0-f210.google.com (mail-fx0-f210.google.com [209.85.220.210]) by mx1.freebsd.org (Postfix) with ESMTP id BFDD78FC08 for ; Sat, 1 Aug 2009 17:37:23 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: by fxm6 with SMTP id 6so649638fxm.43 for ; Sat, 01 Aug 2009 10:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=XkfxmeUkLUabi1SXFGXAWt/bF2h1fCZlrEDiAS6s/LU=; b=hM93WEbsrDQs+iJFw+2cFnIdBZjRYulJgqnTdF8rf9CvK8ytldRmP84qCKloHbFPRg dTMMYAK++ioq74tNpHBbM/xMTaClNrC0iE3KCGZYRriagQL4vh+KyFV/EEqixN/30zKO ez0XUHBQCJADoVI23J3eLw1K8+HVLhx2BKmjo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=AjCm4iQsif/44tKE6X6vdQhKOYVvrCUu3V3RsOqUa9W7BpR4KlhWksgPovKcZ0wytK zXxjE55frT9gOX2oU/g325sGoC1/NNRC2QAErk2lp9hHi1jqS4HlIYrnTbIsB1rJ8c7z CG0KF4BoLhsdhne0Yom4Q8vsgXUC1JimYwNZ4= MIME-Version: 1.0 Sender: asmrookie@gmail.com Received: by 10.223.124.147 with SMTP id u19mr2000271far.28.1249146850759; Sat, 01 Aug 2009 10:14:10 -0700 (PDT) In-Reply-To: <237c27100908010841g5abd3183w479121b2ba7b0c91@mail.gmail.com> References: <237c27100908010841g5abd3183w479121b2ba7b0c91@mail.gmail.com> Date: Sat, 1 Aug 2009 19:14:10 +0200 X-Google-Sender-Auth: ebb0fd75c9b3a52b Message-ID: <3bbf2fe10908011014r2fda9245xc7c5f71fcc544d10@mail.gmail.com> From: Attilio Rao To: Linda Messerschmidt Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: Where have all the vnodes gone? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Aug 2009 17:37:24 -0000 2009/8/1 Linda Messerschmidt : > (Reposted from freebsd-questions due to no replies.) > > With the last few releases, I've noticed a distinct trend toward > disappearing vnodes on one of the machines I look after. > > This machine isn't doing a whole lot. It runs a couple of small web > sites, and once an hour it rsync's some files from one NFS mount to > another, but the rsync doesn't stay running; it restarts every hour > and runs for 10-15 minutes. > > I set it to log the number of vnodes every ten minutes and this is what I got: > > 00:47:59 vfs.numvnodes: 39337 > 00:57:59 vfs.numvnodes: 40568 > 01:07:59 vfs.numvnodes: 44554 > 01:17:59 vfs.numvnodes: 52141 > 01:27:59 vfs.numvnodes: 55713 > 01:37:59 vfs.numvnodes: 58643 > 01:47:59 vfs.numvnodes: 60792 > 01:57:59 vfs.numvnodes: 67130 > 02:07:59 vfs.numvnodes: 76035 > 02:17:59 vfs.numvnodes: 84349 > 02:27:59 vfs.numvnodes: 92187 > 02:37:59 vfs.numvnodes: 98114 > 02:47:59 vfs.numvnodes: 116854 > 02:57:59 vfs.numvnodes: 124842 > 03:07:59 vfs.numvnodes: 164173 > 03:17:59 vfs.numvnodes: 172257 > 03:27:59 vfs.numvnodes: 178388 > 03:37:59 vfs.numvnodes: 183066 > 03:47:59 vfs.numvnodes: 190092 > 03:57:59 vfs.numvnodes: 198322 > 04:07:59 vfs.numvnodes: 204598 > 04:17:59 vfs.numvnodes: 208311 > 04:27:59 vfs.numvnodes: 214207 > 04:37:59 vfs.numvnodes: 221028 > 04:47:59 vfs.numvnodes: 227792 > 04:57:59 vfs.numvnodes: 233214 > 05:07:59 vfs.numvnodes: 240112 > 05:17:59 vfs.numvnodes: 247572 > 05:27:59 vfs.numvnodes: 256090 > 05:37:59 vfs.numvnodes: 262720 > 05:47:59 vfs.numvnodes: 269755 > 05:57:59 vfs.numvnodes: 274986 > 06:07:59 vfs.numvnodes: 279879 > 06:17:59 vfs.numvnodes: 287039 > 06:27:59 vfs.numvnodes: 291984 > 06:37:59 vfs.numvnodes: 294267 > 06:47:59 vfs.numvnodes: 296658 > 06:57:59 vfs.numvnodes: 299086 > 07:07:59 vfs.numvnodes: 301825 > 07:17:59 vfs.numvnodes: 309060 > 07:27:59 vfs.numvnodes: 312955 > 07:37:59 vfs.numvnodes: 317400 > 07:47:59 vfs.numvnodes: 320047 > > At that point the machine crashed with: > > panic: kmem_malloc(16384): kmem_map too small: 334745600 total allocated > > If I don't tune kern.maxvnodes up to the point where it panics, then > eventually it runs out of vnodes and all sorts of stuff gets stuck in > vlruwk. > > The machine in question is running 7.2-RELEASE-p3, but I already > upgraded it from 7.1 trying to get this to go away, so it's a problem > that's been around for awhile. > > My guess is that they're leaking in the kernel somewhere because of > the rsync, because there's just not much else going on, but unless I > can figure out how many vnodes are being used on a per-process basis, > I can't make any headway on proving or disproving that. I do know > that according to fstat, there are only 1000-1500 descriptors open at > any given time, and kern.openfiles ranges 250-500. So I'm just > mystified what the other 300000+ could be. > > Is there any way to figure out where all these vnodes are going? It seems you can reproduce it easilly. Can you please provide KTR traces of KTR_VFS? Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein