Date: Thu, 02 Jan 2003 09:34:26 -0500 From: "Bill Moran" <bill_moran2@hotmail.com> To: myraq@mgm51.com, freebsd-questions@FreeBSD.ORG Subject: Re: Lots of files in a directory Message-ID: <F148vKMbavVQ9WF5knB00001562@hotmail.com>
next in thread | raw e-mail | index | archive | help
>From: "MikeM" <MyRaQ@mgm51.com> > >I'm considering setting up my server as a mirror site for the freedb.org >lookup database. Unfortunately, I've seem to have run into a stumbling >block. The server app requires over 250,000 files in a single directory. >Each file is about 2k in size. > >It was a surprise to me (but probably not to those on this list) that the >file system does not handle that many files in an expeditious manner (I'm >being kind here). > >Is there anything I can do so that the file system works faster with such a >large number of files? I'm looking for an increase in the area of 5 to 1. > For example, the command "rm -rf misc" where "misc" is the directory >containing the 250,000 files takes a couple of hours to run. If "misc" is >my current working directory, and I type "rm *" I get the message that >there are too many arguments being passed into rm. Try the dirhash option in your kernel (as suggested elsewhere). As for commands like rm, you have a few options (as I was recently taught). find(1) appears to handle large numbers of files, thus you can use it as a pipe for just about anything, i.e.: find /path/to/misc -name '*' -print0 | xargs -0 rm (although you don't really need the pipe for rm, but it's an example) I don't know how much of a problem this is for you, but it will work for mv, cp, grep ... just about anything. The quotes around * are important. Also, there is a sysctl (kern.argmax) that you can tweak to increase the length of command line arguments that can be processed, but I don't know how big you'd have to make it to handle 250,000 files! Good luck, Bill _________________________________________________________________ MSN 8 with e-mail virus protection service: 2 months FREE* http://join.msn.com/?page=features/virus To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F148vKMbavVQ9WF5knB00001562>