From owner-freebsd-questions@FreeBSD.ORG Tue Jul 16 10:34:06 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EF6D81A5 for ; Tue, 16 Jul 2013 10:34:06 +0000 (UTC) (envelope-from frank2@fjl.co.uk) Received: from bs1.fjl.org.uk (bs1.fjl.org.uk [84.45.41.196]) by mx1.freebsd.org (Postfix) with ESMTP id 66DD7CA0 for ; Tue, 16 Jul 2013 10:34:06 +0000 (UTC) Received: from [192.168.1.35] (mux.fjl.org.uk [62.3.120.246]) (authenticated bits=0) by bs1.fjl.org.uk (8.14.4/8.14.4) with ESMTP id r6GAXwRZ080493 (version=TLSv1/SSLv3 cipher=DHE-DSS-CAMELLIA256-SHA bits=256 verify=NO) for ; Tue, 16 Jul 2013 11:33:59 +0100 (BST) (envelope-from frank2@fjl.co.uk) Message-ID: <51E52190.7020008@fjl.co.uk> Date: Tue, 16 Jul 2013 11:33:52 +0100 From: Frank Leonhardt User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-questions@freebsd.org Subject: Re: to gmirror or to ZFS References: <4DFBC539-3CCC-4B9B-AB62-7BB846F18530@gmail.com> <976836C5-F790-4D55-A80C-5944E8BC2575@gmail.com> <51E51558.50302@ShaneWare.Biz> In-Reply-To: <51E51558.50302@ShaneWare.Biz> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jul 2013 10:34:07 -0000 On 16/07/2013 10:41, Shane Ambler wrote: > On 16/07/2013 14:41, aurfalien wrote: >> >> On Jul 15, 2013, at 9:23 PM, Warren Block wrote: >> >>> On Mon, 15 Jul 2013, aurfalien wrote: >>> >>>> ... thats the question :) >>>> >>>> At any rate, I'm building a rather large 100+TB NAS using ZFS. >>>> >>>> However for my OS, should I also ZFS or simply gmirror as I've a >>>> dedicated pair of 256GB SSD drives for it. I didn't ask for SSD >>>> sys drives, this system just came with em. >>>> >>>> This is more of a best practices q. >>> >>> ZFS has data integrity checking, gmirror has low RAM overhead. >>> gmirror is, at present, restricted to MBR partitioning due to >>> metadata conflicts with GPT, so 2TB is the maximum size. >>> >>> Best practices... depends on your use. gmirror for the system >>> leaves more RAM for ZFS. >> >> Perfect, thanks Warren. >> >> Just what I was looking for. > > I doubt that you would save any ram having the os on a non-zfs drive as > you will already be using zfs chances are that non-zfs drives would only > increase ram usage by adding a second cache. zfs uses it's own cache > system and isn't going to share it's cache with other system managed > drives. I'm not actually certain if the system cache still sits above > zfs cache or not, I think I read it bypasses the traditional drive cache. > > For zfs cache you can set the max usage by adjusting vfs.zfs.arc_max > that is a system wide setting and isn't going to increase if you have > two zpools. > > Tip: set the arc_max value - by default zfs will use all physical ram > for cache, set it to be sure you have enough ram left for any services > you want running. > > Have you considered using one or both SSD drives with zfs? They can be > added as cache or log devices to help performance. > See man zpool under Intent Log and Cache Devices. > I agree with the sentiment of using the SSD as ZFS cache - it's possibly the only logical use for them. I guess that with 100Tb worth of Winchesters you're not on a very tight budget, and not too tight on RAM for the OS either. If I was going to do this I'd stick with the OS on UFS and a gmirror because I simply don't trust ZFS. This is based on pure prejudice and inexperience. I know how to arrange disks on a UNIX file system for performance - what to use for swap, where tmp files should go and so on. I also know where every file will be, physically, in the event of trouble. And here's the clincher: If the machine blows up I can simply take one of the mirrored drives, slap it in to some new hardware and I've got a very reasonable chance that it'll boot. Can I do this with ZFS? I get the feeling that the answer is an emphatic "maybe". So all things considered, I'd need a good reason not to stick with what I know works reliably and can be recovered in the event of a disaster (UFS), but I'm happy to watch and learn from everyone else's experience!