From owner-freebsd-current@FreeBSD.ORG Tue Jun 15 13:53:36 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E53C916A4CE for ; Tue, 15 Jun 2004 13:53:36 +0000 (GMT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6C8DA43D60 for ; Tue, 15 Jun 2004 13:53:36 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id i5FDr7E8013987; Tue, 15 Jun 2004 08:53:07 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <40CEFF3A.7060202@centtech.com> Date: Tue, 15 Jun 2004 08:52:58 -0500 From: Eric Anderson User-Agent: Mozilla Thunderbird 0.6 (X11/20040520) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Eugene References: <40CEFAA8.00B9A6.08476@e-post02.e-se.ru> In-Reply-To: <40CEFAA8.00B9A6.08476@e-post02.e-se.ru> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-current@freebsd.org Subject: Re: Storing a lot of little files X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jun 2004 13:53:37 -0000 Eugene wrote: >Hello freebsd-current, > > I need to store a lot (hundreds of millions) of very little files (from 8 bytes > to 50K) in my filesystem. > What's the best way to optimize it? Which newfs options can you > recommend me? > Can UFS2 optiomize storage of little files? For example, put some > files in one cluster - so don't let the 8 byte file to occupy a > whole cluster of about 16K? > Is it possible to allocate inodes dynamically (like in ReiserFS?), > bacause I'm not sure I can predict accuratly number of little files, > and inodes not to finish before disk get full. > > Does FreeBSD support file systems except of UFS? > What about ReiserFS, XFS, JFS? > > I've never done 'hundreds of millions', but I've done about 10 million.. I can say that you will definitely need to organize them in a heirarchical directory structure.. You'll want to look at the following options to newfs: -g -h -i and these options to tunefs: -e -f -s Eric -- ------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Talk sense to a fool and he calls you foolish. ------------------------------------------------------------------