From owner-freebsd-fs@FreeBSD.ORG Tue May 29 05:00:41 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [69.147.83.53]) by hub.freebsd.org (Postfix) with ESMTP id 08C7B106567D for ; Tue, 29 May 2012 05:00:41 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 9B0E3151FE3; Tue, 29 May 2012 05:00:40 +0000 (UTC) Message-ID: <4FC457F7.9000800@FreeBSD.org> Date: Mon, 28 May 2012 22:00:39 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Alessio Focardi References: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> In-Reply-To: <1490568508.7110.1338224468089.JavaMail.root@zimbra.interconnessioni.it> X-Enigmail-Version: 1.4.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 05:00:41 -0000 On 5/28/2012 10:01 AM, Alessio Focardi wrote: > So in my case I would have to use -b 4096 -f 512 > > It's an improvement, but still is not ideal: still a big waste with 200 bytes files! Are all of the files exactly 200 bytes? If so that's likely the best you can do. The good news is that it's a big improvement (I've done similar stuff in the past). You'll also want to tweak the -i (inode) value to insure that you have sufficient inodes for the number of files you plan to store. The default is not likely to be adequate for your needs. Don't use journaling of any kind. If your primary access is going to be reads, you should be Ok with softupdates. If you're going to be doing a lot of writing consider no softupdates and async. Of course, all of this assumes that you have a reasonable disk array setup (RAID or similar, power backup, etc.) From the rest of your thread it seems that you have this under control. I think you already referenced this, but you also want to shard the file system so that you don't have any one directory with more than 10k files. Even with dirhash that's still a limitation. You don't have to worry too much if you go a few files over 10k, but you don't want to stretch it much more than that. Make sure you plan for growth over time. hth, Doug -- This .signature sanitized for your protection