From owner-freebsd-stable@FreeBSD.ORG Wed Sep 1 00:21:08 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D5B1A16A4CE for ; Wed, 1 Sep 2004 00:21:08 +0000 (GMT) Received: from ganymede.hub.org (blk-222-46-91.eastlink.ca [24.222.46.91]) by mx1.FreeBSD.org (Postfix) with ESMTP id A69C643D41 for ; Wed, 1 Sep 2004 00:21:08 +0000 (GMT) (envelope-from scrappy@hub.org) Received: by ganymede.hub.org (Postfix, from userid 1000) id 62059374DC; Tue, 31 Aug 2004 21:21:09 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by ganymede.hub.org (Postfix) with ESMTP id 5BB4137406 for ; Tue, 31 Aug 2004 21:21:09 -0300 (ADT) Date: Tue, 31 Aug 2004 21:21:09 -0300 (ADT) From: "Marc G. Fournier" To: freebsd-stable@freebsd.org Message-ID: <20040831205907.O31538@ganymede.hub.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: vnodes - is there a leak? where are they going? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Sep 2004 00:21:09 -0000 I have two servers, both running 4.10 of within a few days (Aug 5 for venus, Aug 7 for neptune) ... both running jail environments ... one with ~60 running, the other with ~80 ... the one with 60 has been running for ~25 days now, and is at the border of running out of vnodes: Aug 31 20:58:00 venus root: debug.numvnodes: 519920 - debug.freevnodes: 11058 - debug.vnlru_nowhere: 256463 - vlrup Aug 31 20:59:01 venus root: debug.numvnodes: 519920 - debug.freevnodes: 13155 - debug.vnlru_nowhere: 256482 - vlrup Aug 31 21:00:03 venus root: debug.numvnodes: 519920 - debug.freevnodes: 13092 - debug.vnlru_nowhere: 256482 - vlruwt while the other one has been up for ~1 days, but is using alot less, for more processes: Aug 31 20:58:00 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 208655 - debug.vnlru_nowhere: 0 - vlruwt Aug 31 20:59:00 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 208602 - debug.vnlru_nowhere: 0 - vlruwt Aug 31 21:00:03 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 208319 - debug.vnlru_nowhere: 0 - vlruwt I've tried shutting down all of the VMs on venus, and umount'd all of the unionfs mounts, as well as the one nfs mount we have ... the above #s are after the VMs (and mounts are recreated ... Now, my understanding of the vnodes is that for every file opened, a vnode is created ... in my case, since I'm using unionfs, there are two vnodes per file ... if it possible that there are 'stale' vnodes that aren't being freed up? Is there some way of 'viewing' the vnode structure? For instance, fstat shows: venus# fstat | wc -l 19531 So, obviously it isn't just open files that I'm dealing with here, for even if I double that, that is nowhere near 519920 ... So, where else are the vnodes going? Is there a 'leak'? What can I look at to try and narrow this down / provide more information? Even some way of determining a specific process that is sucking back alot of them, to move that to a different machine ... ? Help? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664