From owner-freebsd-fs@FreeBSD.ORG Tue May 29 07:04:36 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D36B8106564A for ; Tue, 29 May 2012 07:04:36 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id 44DD98FC16 for ; Tue, 29 May 2012 07:04:35 +0000 (UTC) Received: from server.rulingia.com (c220-239-254-65.belrs5.nsw.optusnet.com.au [220.239.254.65]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id q4T74Yke050015 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 29 May 2012 17:04:35 +1000 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from aspire.rulingia.com (aspire.rulingia.com [192.168.123.161]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id q4T74Rtn046867 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 29 May 2012 17:04:27 +1000 (EST) (envelope-from peter@rulingia.com) Received: from aspire.rulingia.com (localhost [127.0.0.1]) by aspire.rulingia.com (8.14.5/8.14.5) with ESMTP id q4T6c5Qx038725 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 29 May 2012 16:38:10 +1000 (EST) (envelope-from peter@aspire.rulingia.com) Received: (from peter@localhost) by aspire.rulingia.com (8.14.5/8.14.5/Submit) id q4T6c48D038724; Tue, 29 May 2012 16:38:04 +1000 (EST) (envelope-from peter) Date: Tue, 29 May 2012 16:38:03 +1000 From: Peter Jeremy To: Alessio Focardi Message-ID: <20120529063803.GG2675@aspire.rulingia.com> References: <588211375.4794.1338210497900.JavaMail.root@zimbra.interconnessioni.it> <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="iAL9S67WQOXgEPD9" Content-Disposition: inline In-Reply-To: <2134924725.5040.1338211317460.JavaMail.root@zimbra.interconnessioni.it> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: Millions of small files: best filesystem / best options X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2012 07:04:36 -0000 --iAL9S67WQOXgEPD9 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2012-May-28 15:21:57 +0200, Alessio Focardi wrote: >I'm looking for some advice to efficiently pack millions of small files (2= 00 bytes or less) over a freebsd fs. "Millions of files" isn't an issue - any self-respecting filesystem will manage that. The "200 bytes or less" is a real problem. The best you will be able to do is a 4K/512 UFS1 filesystem (UFS1 because the inodes are half the size) - that gives you a minimum of 640 bytes (plus directory entry) per file (it would be 768 bytes for UFS2 and 1536 or 2048 bytes for ZFS). I would suggest that no normal filesystem will be a good match for your requirements. IMHO, you would be better off storing your data as an array of "file+metadata" objects within a single physical file, together with some sort of index structure that suits your "file" names. --=20 Peter Jeremy --iAL9S67WQOXgEPD9 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAk/EbsoACgkQ/opHv/APuIdrtACgs5rseqWRzW/8nsc+XxE3SCM6 fOEAoK3c77JusLQomJ+YygpKhYCh9XO8 =gO6N -----END PGP SIGNATURE----- --iAL9S67WQOXgEPD9--