From owner-freebsd-fs@FreeBSD.ORG Sat Sep 4 00:33:35 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6925610656B5 for ; Sat, 4 Sep 2010 00:33:35 +0000 (UTC) (envelope-from prvs=18630f88c7=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 0622A8FC13 for ; Sat, 4 Sep 2010 00:33:34 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Sat, 04 Sep 2010 01:22:47 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 04 Sep 2010 01:22:47 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 by mail1.multiplay.co.uk (MDaemon PRO v10.0.4) with ESMTP id md50011181479.msg for ; Sat, 04 Sep 2010 01:22:47 +0100 X-Authenticated-Sender: Killing@multiplay.co.uk X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=18630f88c7=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@FreeBSD.ORG Message-ID: <5DB6E7C798E44D33A05673F4B773405E@multiplay.co.uk> From: "Steven Hartland" To: Date: Sat, 4 Sep 2010 01:22:47 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 Cc: Subject: zfs very poor performance compared to ufs due to lack of cache? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Sep 2010 00:33:35 -0000 When upgrading from 8.0 on our stream server to 8.1 we decided to go for zfs to eliminate the costly fsck times should we experience any unexpected reboots on the machine as it has a sizable RAID of 1.6TB. After doing this all seemed good till after our latest event which generated a significant amount of interest and hence the stream server started to get quite a few requests. Basic install is 8.1 amd64 on a dual 2.8 Xeon with 4GB RAM and areca controller with 6 disk in RAID 6. On that the machine runs nginx with the mp4 module to sudo stream files. Previously this machine didn't have any problem dealing with significant numbers of clients as long as the clients where largely playing the same files ( which are typically quite large 1GB+ ) however after the update to zfs it seems to be really struggling with disk IO, maxing out the disk for large periods of time even when all the "active" files should be able to fit easily into ram. After digging around it seems that the default config is very conservative, not the case with ufs which seems to use all free ram as cache which is exactly what we want as nginx uses so little ram there's nothing else for it to do. So now we've added the following to loader.conf vfs.zfs.cache_flush_disable=1 # Battery Backed RAID vfs.zfs.prefetch_disable=0 # Large Streaming files seems like this would be good vm.kmem_size="6G" # Physical Memory * 1.5 as recommend in several places vfs.zfs.arc_max="3584M" # Most of the available RAM I thought this would help but apparently not, zfs still only seems to want to use a minimal amount of ram for cache as shown by arc_summary ARC Size: Current Size: 12.50% 447.97M (arcsize) Target Size: (Adaptive) 12.50% 448.00M (c) Min Size (Hard Limit): 12.50% 448.00M (c_min) Max Size (High Water): ~8:1 3584.00M (c_max) For reference top shows:- Mem: 42M Active, 3129M Inact, 565M Wired, 3188K Cache, 19M Buf, 203M Free Swap: 4096M Total, 1180K Used, 4095M Free So what are we missing and how to we get zfs to perform like ufs and use all free ram as cache? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.