Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 29 May 2012 09:37:19 -0700
From:      Julian Elischer <julian@freebsd.org>
To:        Alessio Focardi <alessiof@gmail.com>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: Millions of small files: best filesystem / best options
Message-ID:  <4FC4FB3F.8030203@freebsd.org>
In-Reply-To: <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it>
References:  <49722655.1520.1338282954302.JavaMail.root@zimbra.interconnessioni.it>

next in thread | previous in thread | raw e-mail | index | archive | help
On 5/29/12 2:15 AM, Alessio Focardi wrote:
>>> I ran a Usenet server this way for quite a while with fairly good
>>> results, though the average file size was a bit bigger, about 2K or
>>> so.
>>> I found that if I didn't use "-o space" that space optimization
>>> wouldn't
>>> kick in soon enough and I'd tend to run out of full blocks that
>>> would be
>>> needed for larger files.
> Fragmentation is not a problem for me, mostly I will have a write once-read many situation, still is not clear to me if "-o space" works in the constraints of the block/fragment ratio, that in my case it would still mean that I will have to use a 512 bytes subblock for every 200 byte files.
>
> ps
>
> really thank you for all of your help!
>
>
> Alessio Focardi
> ------------------
>
>
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
>
Maybe try use something other than a filesystem. The problem you will 
have is that the physical media is going to be formatted to 512 bytes 
or larger so every operation will be a read/modify/write if you try do 
to much packing..  As others have said it makes a big difference if 
you are talking about 20MB of data or 200GB of data too..  If it fits 
in ram, you could use 'mailbox format' and pack them all in one file 
and then just load it into ram when you need to access it.

You could always buy one of our flash cards in key-value-store mode if 
you need 1.3 TB of 512 byte values at 400,000 values per 
second(read/write), but the pricetag might be a bit scary.. :-) (10k-ish)






Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4FC4FB3F.8030203>