From owner-cvs-all Wed Oct 3 22:13:22 2001 Delivered-To: cvs-all@freebsd.org Received: from peter3.wemm.org (c1315225-a.plstn1.sfba.home.com [24.14.150.180]) by hub.freebsd.org (Postfix) with ESMTP id 4108F37B401; Wed, 3 Oct 2001 22:13:10 -0700 (PDT) Received: from overcee.netplex.com.au (overcee.wemm.org [10.0.0.3]) by peter3.wemm.org (8.11.0/8.11.0) with ESMTP id f945D9M55360; Wed, 3 Oct 2001 22:13:09 -0700 (PDT) (envelope-from peter@wemm.org) Received: from wemm.org (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id BFAF23808; Wed, 3 Oct 2001 22:13:09 -0700 (PDT) (envelope-from peter@wemm.org) X-Mailer: exmh version 2.3.1 01/18/2001 with nmh-1.0.4 To: Matt Dillon Cc: cvs-committers@FreeBSD.org, cvs-all@FreeBSD.org Subject: Re: cvs commit: src/sys/kern vfs_cache.c vfs_subr.c src/sys/sys vnode.h In-Reply-To: <200110010433.f914XZd11860@freefall.freebsd.org> Date: Wed, 03 Oct 2001 22:13:09 -0700 From: Peter Wemm Message-Id: <20011004051309.BFAF23808@overcee.netplex.com.au> Sender: owner-cvs-all@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Matt Dillon wrote: > dillon 2001/09/30 21:33:35 PDT > I have also determined through testing that the real problem with numvnodes > getting too large is due to the VM Page cache preventing the vnode from > being reclaimed. The directory stuff made only a tiny dent relative > to Poul's original code, enough so that some tests succeeded. But tests > with several million small files show that the bigger problem is the VM > Page cache. This will have to be addressed by a future commit. I ran into this problem several times in the last few days trying to copy a 20G file tree from a raid to a disk and back again. I ended up "solving" the problem by having a script that did this: cd /home/cd/freebsd-iso/ while 1 tar cvf /dev/null . sleep 300 end This read about 2.7GB of freebsd cdrom .iso's and flused the VM page cache. This caused about 260,000 vnodes to be freed shortly thereafter instead of running out of ffsnodes. The machine had 3G of ram at the time and it locked up every time I tried to copy the archive. It was a RELENG_4 box without vmiodirenable (but turning it on made no difference). I believe this is the same problem that we're seeing at yahoo. On this particular box, while it has been idle for a while: 197126 desiredvnodes 214634 numvnodes 5158 freevnodes peter@thunder[10:02pm]~-102> vmstat -m | grep ' FFS' FFS node214626 53657K 53657K102400K 3268258 0 0 256 ie: it has used over 50% of its M_FFSNODE limit. This is probably over-simplistic, but can we more aggressively reclaim older vnodes (and consequently freeing their pages) and recycle them when we're over the desiredvnodes limit? Having to blow away the enire VM page cache in order to make the vnodes recyclable is a bit silly. And even then they are never "freed", they just go into the free list but keep their associated ffs inode etc. Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe cvs-all" in the body of the message