From owner-freebsd-fs@FreeBSD.ORG Tue May 29 16:37:07 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1FFFE106564A for ; Tue, 29 May 2012 16:37:07 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id E2C188FC1A for ; Tue, 29 May 2012 16:37:06 +0000 (UTC) Received: from julian-mac.elischer.org (c-67-180-24-15.hsd1.ca.comcast.net [67.180.24.15]) (authenticated bits=0) by vps1.elischer.org (8.14.5/8.14.5) with ESMTP id q4TGb5eh095379 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 29 May 2012 09:37:06 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <4FC4FB3F.8030203@freebsd.org> Date: Tue, 29 May 2012 09:37:19 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20 MIME-Version: 1.0 To: Alessio Focardi References: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 16:37:07 -0000 On 5/29/12 2:15 AM, Alessio Focardi wrote: >>> I ran a Usenet server this way for quite a while with fairly good >>> results, though the average file size was a bit bigger, about 2K or >>> so. >>> I found that if I didn't use "-o space" that space optimization >>> wouldn't >>> kick in soon enough and I'd tend to run out of full blocks that >>> would be >>> needed for larger files. > Fragmentation is not a problem for me, mostly I will have a write once-read many situation, still is not clear to me if "-o space" works in the constraints of the block/fragment ratio, that in my case it would still mean that I will have to use a 512 bytes subblock for every 200 byte files. > > ps > > really thank you for all of your help! > > > Alessio Focardi > ------------------ > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > Maybe try use something other than a filesystem. The problem you will have is that the physical media is going to be formatted to 512 bytes or larger so every operation will be a read/modify/write if you try do to much packing.. As others have said it makes a big difference if you are talking about 20MB of data or 200GB of data too.. If it fits in ram, you could use 'mailbox format' and pack them all in one file and then just load it into ram when you need to access it. You could always buy one of our flash cards in key-value-store mode if you need 1.3 TB of 512 byte values at 400,000 values per second(read/write), but the pricetag might be a bit scary.. :-) (10k-ish)