From owner-freebsd-questions@FreeBSD.ORG Tue Mar 29 01:44:24 2005 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 06B2F16A4D0 for ; Tue, 29 Mar 2005 01:44:22 +0000 (GMT) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8A4B843D39 for ; Tue, 29 Mar 2005 01:44:21 +0000 (GMT) (envelope-from mcbrune@gmail.com) Received: by wproxy.gmail.com with SMTP id 69so379349wra for ; Mon, 28 Mar 2005 17:44:20 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=rf034rfSYzhRRRw/kZClvtIugYvdYJmPUtDEuDN8okLkz81CxjQBbt2kmrLnzyhj33xNxRQdUina4KAmE1lL22lsl0uuFqn52O5ss40QnefExIncHZs4XInC8A+M2MJegxO0N0sx3gCHkXMRsrxzRmDG7j0LYWd0XdvhGSzGo88= Received: by 10.54.73.3 with SMTP id v3mr214033wra; Mon, 28 Mar 2005 17:44:20 -0800 (PST) Received: by 10.54.71.20 with HTTP; Mon, 28 Mar 2005 17:44:20 -0800 (PST) Message-ID: <5627053705032817444c6c4df4@mail.gmail.com> Date: Mon, 28 Mar 2005 19:44:20 -0600 From: Corey Brune To: David Pratt In-Reply-To: <6EDD6E31-9FF1-11D9-8CB8-000A27B3B070@eastlink.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit References: <20050324193456.GB31083@Grumpy.DynDNS.org> <6EDD6E31-9FF1-11D9-8CB8-000A27B3B070@eastlink.ca> cc: FreeBSD-Questions@freebsd.org Subject: Re: Max files in unix folder from PIL process X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Corey Brune List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Mar 2005 01:44:24 -0000 On Mon, 28 Mar 2005 21:25:28 -0400, David Pratt wrote: > Hi. I am creating a python application that uses PIL to generate > thumbnails and sized images. It is beginning to look the volume of > images will be large. This has got me to thinking. Is there a number > that Unix can handle in a single directory. I am using FreeBSD4.x at > the moment. I am thinking the number could be as high 500,000 images in > a single directory but more likely in the range of 6,000 to 30,000 for > most. I did not want to store these in Postgres. I will most likely to > break these into directories by size ie. thumbnail, small, medium, > large, etc. . That will at least take it down by a factor of the > number of sizes used but still the possibility of a very large number > (maximum to perhaps 100,000 or more) There is really no other way that > I can think of to categorize these at the moment. Should this pose a > problem on the filesystem? How will it affect the use of Unix tools? > Will there be access problems that affect speed? This is unchartered > territory for me so hope someone who has been there, done that can > provide some of what they learned from experience. Many thanks. > > Regards, > David > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" > I've had apps that had over 800k subdirectories and files. As long as you know the filename, performance will not be an issue. However, if you don't know the full path, then you may want to either redesign the app or consider storing everything in the DB. The apps that I've seen had the full path stored in the database, and the image on a file server. I've also had apps that stored everything in the database, and that turned out to be a nightmare. Eventually, we moved everything from the DB to the filesystem. Corey