Date: Sat, 4 Sep 2010 01:22:47 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: <freebsd-fs@FreeBSD.ORG> Subject: zfs very poor performance compared to ufs due to lack of cache? Message-ID: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk>
next in thread | raw e-mail | index | archive | help
When upgrading from 8.0 on our stream server to 8.1 we decided to go
for zfs to eliminate the costly fsck times should we experience
any unexpected reboots on the machine as it has a sizable RAID of
1.6TB.
After doing this all seemed good till after our latest event which
generated a significant amount of interest and hence the stream
server started to get quite a few requests.
Basic install is 8.1 amd64 on a dual 2.8 Xeon with 4GB RAM and
areca controller with 6 disk in RAID 6.
On that the machine runs nginx with the mp4 module to sudo stream
files.
Previously this machine didn't have any problem dealing with
significant numbers of clients as long as the clients where largely
playing the same files ( which are typically quite large 1GB+ )
however after the update to zfs it seems to be really struggling
with disk IO, maxing out the disk for large periods of time
even when all the "active" files should be able to fit easily
into ram.
After digging around it seems that the default config is very
conservative, not the case with ufs which seems to use all free
ram as cache which is exactly what we want as nginx uses so little
ram there's nothing else for it to do.
So now we've added the following to loader.conf
vfs.zfs.cache_flush_disable=1 # Battery Backed RAID
vfs.zfs.prefetch_disable=0 # Large Streaming files seems like this
would be good
vm.kmem_size="6G" # Physical Memory * 1.5 as recommend in several
places
vfs.zfs.arc_max="3584M" # Most of the available RAM
I thought this would help but apparently not, zfs still only seems to
want to use a minimal amount of ram for cache as shown by arc_summary
ARC Size:
Current Size: 12.50% 447.97M (arcsize)
Target Size: (Adaptive) 12.50% 448.00M (c)
Min Size (Hard Limit): 12.50% 448.00M (c_min)
Max Size (High Water): ~8:1 3584.00M (c_max)
For reference top shows:-
Mem: 42M Active, 3129M Inact, 565M Wired, 3188K Cache, 19M Buf, 203M Free
Swap: 4096M Total, 1180K Used, 4095M Free
So what are we missing and how to we get zfs to perform like ufs
and use all free ram as cache?
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5DB6E7C798E44D33A05673F4B773405E>