Date: Fri, 14 Nov 2008 22:58:24 -0600 From: "Rick C. Petty" <rick-freebsd2008@kiwi-computer.com> To: Peter Jeremy <peterjeremy@optushome.com.au> Cc: freebsd-fs@freebsd.org Subject: Re: Filesystem size and free space Message-ID: <20081115045824.GA23464@keira.kiwi-computer.com> In-Reply-To: <20081115014203.GE51761@server.vk2pj.dyndns.org> References: <491D5296.3000600@bsd.ee> <20081115014203.GE51761@server.vk2pj.dyndns.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Note, the details in this message are meant for the original poster. On Sat, Nov 15, 2008 at 12:42:03PM +1100, Peter Jeremy wrote: > > In the case of UFS2, the size shown as x-Blocks reflects the size of > the underlying media, less a free space allowance: 8% [not 10%] by > default - see the -m option of tunefs for details of this and why it > exists. Out of this, UFS2 allocates file and direcory data blocks, > file metadata and filesystem metadata. By default, data blocks are > 16KB with 2KB fragments. Each file or directory needs 256 bytes of > metadata (its inode). I can't quickly find the size of the filesystem > metadata but estimate it is <<1% of the filesystem size. Not true. With the current defaults, and not including the 320k reserved at the beginning (for bootblocks, etc.) nor the few MB at the end to align with a cylinder boundary, UFS2 takes around 3% space for its metadata. Almost all of this is inode allocation. Most people don't need nearly this many inodes, but newfs(8) chooses too many instead of too few because running out of inodes is more frustrating (I've done it before and I agree). But you can change this default, and I do this for all my filesystems. Specifying a low inode density can get you into trouble but can reduce the metadata consumption to around 0.3% of the filesystem size. I haven't done the math, but but it seems that NTFS uses about 12.5% (at best) for storing filesystem metadata. Another big difference is that NTFS slows down considerably at about 75% full whereas UFS and UFS2 perform very well to just past 92%. Also NTFS does have much flexibility in being able to control the amount of filesystem metadata. With the price of drives nowadays, I find complaints about metadata waste particularly annoying. Still, I suggest that the OP should use the inode density parameter to newfs if if insisting that UFS wastes too much space. > - As others have suggested, reducing minfree will help remove the negative > free space. Be careful doing this unless your filesystem is write-once. I wouldn't bother. The performance loss is so great that you're better off buying a larger drive. I have gone into the minfree threshold before but it still beats a similarly-full NTFS partition in terms of performance. > - If you have a few very large files, rebuild the filesystem with fewer > inodes (large '-i' parameter to newfs) and maybe a bigger blocksize. If you know the sizes and numbers of files in advance, it's easy to do the math here. Play with "newfs -N -i <number>" until the number of cylinder groups times the number of inodes (per cylinder group, in case the output is not clear) is higher than the total number of files plus directories you will be storing. I always add an extra margin just to be safe. One warning about using this option-- if you ever intend to use growfs(8), be warned that growfs does not have a -i option nor does it account for the number of inodes you previously specified. It's easy to push the numbers such that a growfs will actually reduce your free space below the -8% and thus fail. I don't see much point in using growfs in general; you either are migrating a volume to a large drive (in which case you're better off newfs-ing and using rsync) or you're trying to fiddle with an existing drive that is probably too small for your needs. I find growfs more useful for working with keyfobs or md(4) devices. > - If you have lots of small files, you might be better off with an 8K/1K > filesystem and maybe even UFS1 (which has a smaller inode size). If you're planning on fiddling with the block and fragment sizes, I would do this before adjusting the -i option, since both affect the outcome of the inode density. I find these parameters harder to configure. You need to know your typical file size (not just average size). If your smaller files are much smaller than the block size, then it makes sense to lower these sizes. If you have fewer but larger files, it might make sense to increase them. It helps to understand how the files are allocated on the filesystem. A file always allocates full blocks (each block being 8 fragments) to store its data except for the last block. If a full block isn't needed, the file allocates the number of fragments needed, leaving the remaining fragments of that block to be used by the last block of another file. This really helps when you have a lot of small files but not as much when your files are large. Another thing I recommend if you have (or are planning to have) a number of filesystems is to keep track of which newfs parameters you used to make each file system. This helps you in planning future filesystems and helps you recreate the original filesystems (in case of a restore from backup or if you plan to move a filesystem to another disk). For example, I created a filesystem to store some videos I recorded from VHS and estimated about 1000 inodes were needed, with an average file size of 300 MB. I created this filesystem using "newfs -U -f 8192 -b 65536 -i 314572800". I still got many more inodes than I needed but consumed less space with metadata. HTH, -- Rick C. Petty
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081115045824.GA23464>