From owner-freebsd-questions Thu Jan 2 6:35:53 2003 Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 35EC437B401 for ; Thu, 2 Jan 2003 06:35:52 -0800 (PST) Received: from hotmail.com (f148.law15.hotmail.com [64.4.23.148]) by mx1.FreeBSD.org (Postfix) with ESMTP id D33BB43EC2 for ; Thu, 2 Jan 2003 06:35:51 -0800 (PST) (envelope-from bill_moran2@hotmail.com) Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Thu, 2 Jan 2003 06:34:26 -0800 Received: from 66.132.12.15 by lw15fd.law15.hotmail.msn.com with HTTP; Thu, 02 Jan 2003 14:34:26 GMT X-Originating-IP: [66.132.12.15] From: "Bill Moran" To: myraq@mgm51.com, freebsd-questions@FreeBSD.ORG Subject: Re: Lots of files in a directory Date: Thu, 02 Jan 2003 09:34:26 -0500 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: X-OriginalArrivalTime: 02 Jan 2003 14:34:26.0801 (UTC) FILETIME=[0D3B7E10:01C2B26C] Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG >From: "MikeM" > >I'm considering setting up my server as a mirror site for the freedb.org >lookup database. Unfortunately, I've seem to have run into a stumbling >block. The server app requires over 250,000 files in a single directory. >Each file is about 2k in size. > >It was a surprise to me (but probably not to those on this list) that the >file system does not handle that many files in an expeditious manner (I'm >being kind here). > >Is there anything I can do so that the file system works faster with such a >large number of files? I'm looking for an increase in the area of 5 to 1. > For example, the command "rm -rf misc" where "misc" is the directory >containing the 250,000 files takes a couple of hours to run. If "misc" is >my current working directory, and I type "rm *" I get the message that >there are too many arguments being passed into rm. Try the dirhash option in your kernel (as suggested elsewhere). As for commands like rm, you have a few options (as I was recently taught). find(1) appears to handle large numbers of files, thus you can use it as a pipe for just about anything, i.e.: find /path/to/misc -name '*' -print0 | xargs -0 rm (although you don't really need the pipe for rm, but it's an example) I don't know how much of a problem this is for you, but it will work for mv, cp, grep ... just about anything. The quotes around * are important. Also, there is a sysctl (kern.argmax) that you can tweak to increase the length of command line arguments that can be processed, but I don't know how big you'd have to make it to handle 250,000 files! Good luck, Bill _________________________________________________________________ MSN 8 with e-mail virus protection service: 2 months FREE* http://join.msn.com/?page=features/virus To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message