Date: Wed, 10 Mar 2010 13:20:09 -0500 From: Nathan Vidican <nathan@vidican.com> To: Janos Dohanics <web@3dresearch.com> Cc: User Questions <freebsd-questions@freebsd.org> Subject: Re: Directory with large number of files Message-ID: <795fc2b81003101020q3a0a73e6u5b43ec4dc1da04ce@mail.gmail.com> In-Reply-To: <201003101151.50738.web@3dresearch.com> References: <201003101151.50738.web@3dresearch.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In some passed projects of mine where large quantities of image files had to be referenced I used a hierarchical tree structure based on the filename. For example: instead of '/disk/images/0018391.jpg' I used '/disk/images/10000/8/3/91.jpg' instead of '/disk/images/abcdsomename.jpg' I used '/disk/images/a/b/cdsomename.jpg' In most cases it was to store large amounts of small image files to reference from web-apps and such with a database key. Usually I tried to work either numerically or alphabetically scaling to the database itself. This also allows you to (if needed) use multiple disks/mount-points down the line if the scale grows (ie: move /disk/images/a or /disk/images/10000 to a dedicated disk/mount if room runs out or for performance reasons). Not sure on the system-side what the advantages versus using a single large directory will be - but I noted it was a much easier way to maintain things and scaled very well (in one case had a directory with well over a million images tied to a database of a product catalog, we broke it down by department/section and then by numeric key and eventually had to dedicate larger/faster storage to certain sections thus off-loading the work from all-in-one and requiring minimal effort to do so). Food for thought anyways - hope it helps. -- Nathan Vidican nathan@vidican.com On Wed, Mar 10, 2010 at 11:51 AM, Janos Dohanics <web@3dresearch.com> wrote: > I have a directory with over 180,000 files in 4 subdirectories. Over > 170,000 > of the files are in one of the subdirectories. The files are image files, > mostly <50K. The machine has 2 GB RAM. > > What would be a reasonable setting for vfs.ufs.dirhash_maxmem? Is there a > disadvantage to setting vfs.ufs.dirhash_maxmem too high? > > Is there any other setting to change to find files faster in a directory > with > such large number of files? > > Thank you for your advice. > > -- > Janos Dohanics > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?795fc2b81003101020q3a0a73e6u5b43ec4dc1da04ce>