From owner-freebsd-current@FreeBSD.ORG  Sat Sep  4 16:56:23 2004
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP
	id ED15F16A4D1; Sat,  4 Sep 2004 16:56:22 +0000 (GMT)
Received: from ganymede.hub.org (blk-222-46-91.eastlink.ca [24.222.46.91])
	by mx1.FreeBSD.org (Postfix) with ESMTP
	id A1E0443D31; Sat,  4 Sep 2004 16:56:22 +0000 (GMT)
	(envelope-from scrappy@hub.org)
Received: by ganymede.hub.org (Postfix, from userid 1000)
	id 55A8D38608; Sat,  4 Sep 2004 13:29:50 -0300 (ADT)
Received: from localhost (localhost [127.0.0.1])
	by ganymede.hub.org (Postfix) with ESMTP id 51F6F33C8A;
	Sat,  4 Sep 2004 13:29:50 -0300 (ADT)
Date: Sat, 4 Sep 2004 13:29:50 -0300 (ADT)
From: "Marc G. Fournier" <scrappy@hub.org>
To: Julian Elischer <julian@elischer.org>
In-Reply-To: <41394D0B.1050004@elischer.org>
Message-ID: <20040904131131.A812@ganymede.hub.org>
References: <20040901151405.G47186@ganymede.hub.org>
 <20040901200257.GA92717@afields.ca><41365746.2030605@samsco.org>
 <20040902013534.GD9327@afields.ca> <20040901224632.O72978@ganymede.hub.org>
 <20040904004706.O812@ganymede.hub.org> <41394D0B.1050004@elischer.org>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
cc: Allan Fields <bsd@afields.ca>
cc: freebsd-stable@freebsd.org
cc: freebsd-current@freebsd.org
Subject: Re: vnode leak in FFS code ... ?
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 04 Sep 2004 16:56:23 -0000

On Fri, 3 Sep 2004, Julian Elischer wrote:

> Marc G. Fournier wrote:
>> 
>> Just as a followup to this ... the server crashed on Thursday night around 
>> 22:00ADT, only just came back up after a very long fsck ... with all 62 VMs 
>> started up, and 1008 processes running, vnodes currently look like:
>
> are you using nullfs at all on your vms?

No, I stop'd using that over a year ago, figuring that it was exasperating 
the problems we were having back then ... the only thing we did use nullfs 
at that time was so that we could 'identify' which files were specific to 
a VM, vs a file on the template ... we moved to using nfs to do the same 
thing ...

The only thing we use is unionfs, and nfs ...

Basically, we do a 'mount_union -b <template> <vm>', where template is a 
shared file system containing common applications, in order to reduce 
overall disk space being used by each client.  So, for instance, on one of 
our servers we have a 'template' VM that, when we need to add/upgrade an 
application, we start up, log into and install from ports ... then, we 
rsync that template to the live server(s) so that those apps are available 
within all VMs ...

We then use NFS to mount the 'base' file system for each VM, that contains 
only the changed files that are specific to the VM (ie. config files, any 
apps the client happens to have installed, etc) and use that to determine 
storage usage ...

there is only one NFS mount point, that covers the whole file system, we 
dont do a mount per VM or anything like that ...

So, in the case of the system that has risen quite high on the vnode 
count, with 60 VMs ... there would be:

5 UFS mounts
 	- /, /var, /tmp, /usr and /vm
 	- /vm is where the virtual machines run off of
1 NFS mount
60 UNIONFS mount points
 	- one for each VM
60 procfs mount points
 	- one for each VM

Thanks to the work that David and Tor put in last summer on vnlru, this 
works quite well, with the occasional crash when a 'fringe bug' gets 
tweaked ... our record uptime on a server, in this configuration, so far 
is 106days ...

The part that hurts the most is that the longer the server is up and 
running, the greater the chance of having a 12+hr fsck run due to all the 
ZERO LENGTH DIRECTORYs :(

Whenever I get a good core dump, I try and post a report to GNaTs, but 
between everyone focusing on 5.x, and those crying "unionfs is broken", 
they tend to sit in limbo ... altho most of the bugs I am able to find 
most likely exist in 5.x code as well, and fixing them would go one more 
step towards improving unionfs ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664