From owner-freebsd-performance@FreeBSD.ORG Wed May 6 15:10:24 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB2AD106571E; Wed, 6 May 2009 15:10:24 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from mail.potentialtech.com (internet.potentialtech.com [66.167.251.6]) by mx1.freebsd.org (Postfix) with ESMTP id 8A8188FC12; Wed, 6 May 2009 15:10:24 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from vanquish.ws.pitbpa0.priv.collaborativefusion.com (pr40.pitbpa0.pub.collaborativefusion.com [206.210.89.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.potentialtech.com (Postfix) with ESMTPSA id 14935EBC0A; Wed, 6 May 2009 11:10:23 -0400 (EDT) Date: Wed, 6 May 2009 11:10:22 -0400 From: Bill Moran To: Olivier Mueller Message-Id: <20090506111022.05d06f1a.wmoran@potentialtech.com> In-Reply-To: <1241616121.16418.109.camel@ompc.insign.local> References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <1241616121.16418.109.camel@ompc.insign.local> X-Mailer: Sylpheed 2.6.0 (GTK+ 2.14.7; i386-portbld-freebsd7.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data (4 million files) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 15:10:25 -0000 In response to Olivier Mueller : > > Yes, it is one of the best options. My initial goal was to delete all > files older than N days by cron (find | xargs | rm, etc.), but if each > cronjob takes 2 hours (and takes so much cpu time), it's probably not > the best way. > > I'll make some more tests on an test-server later this week and speak > with the devs. Thanks again for your very constructive feedback! Based on your comments here, it really sounds like your devs need to implement some sort of cache cleaning algo into their code. If it's just deleting the oldest files, then you could probably run it far more frequently if you simply created a new cache directory each hour, and deleted the previous one. Honestly, I'm really confused -- if you can just throw away the cache each night, then why are you caching to begin with? If you just need temp files, why doesn't the app clean up its temp files when it's done with them? If you have access to the developers, I think you'll be able to come up with a much better solution by working with them. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/