From owner-freebsd-current@FreeBSD.ORG Sun Jan 6 21:32:47 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4F5516A419; Sun, 6 Jan 2008 21:32:47 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B703913C4D9; Sun, 6 Jan 2008 21:32:46 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <478148FD.20605@FreeBSD.org> Date: Sun, 06 Jan 2008 22:32:45 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Scott Long References: <20080106141157.I105@fledge.watson.org> <47810DE3.3050106@FreeBSD.org> <478119AB.8050906@FreeBSD.org> <47814160.4050401@samsco.org> In-Reply-To: <47814160.4050401@samsco.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-current@freebsd.org, Ivan Voras Subject: Re: ZFS honesty X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jan 2008 21:32:48 -0000 Scott Long wrote: > Kris Kennaway wrote: >> Ivan Voras wrote: >>> Kris Kennaway wrote: >>>> Ivan Voras wrote: >>>>> Robert Watson wrote: >>>>> >>>>>> I'm not sure if anyone has mentioned this yet in the thread, but >>>>>> another thing worth taking into account in considering the >>>>>> stability of ZFS is whether or not Sun considers it a production >>>>>> feature in Solaris. Last I heard, it was still considered an >>>>>> experimental feature there as well. >>>>> >>>>> Last I heard, rsync didn't crash Solaris on ZFS :) >>>> >>>> [Citation needed] >>> >>> I can't provide citation about a thing that doesn't happen - you >>> don't hear things like "oh and yesterday I ran rsync on my Solaris >>> with ZFS and *it didn't crash*!" often. >>> >>> But, with some grains of salt taken, consider this Google results: >>> >>> * searching for "rsync crash solaris zfs": 790 results, most of them >>> obviously irrelevant >>> * searching for "rsync crash freebsd zfs": 10,800 results; a small >>> number of the results is from this thread, some are duplicates, but >>> it's a large number in any case. >>> >>> I feel that the number of Solaris+ZFS installations worldwide is >>> larger than that of FreeBSD+ZFS and they've had ZFS longer. >> >> Almost all Solaris systems are 64 bit. >> >> Kris > > So, let's be honest here. ZFS is simply unreliable on FreeBSD/i386. > There are things that you can do mitigate the problems, and in certain > well controlled environments you might be able to make it work well > enough for your needs. But as a general rule, don't expect it to work > reliably, period. This is backed up by Sun's own recommendation to not > run it on 32-bit Solaris. > > But let's also be honest about ZFS in the 64-bit world. There is ample > evidence that ZFS basically wants to grow unbounded in proportion to the > workload that you give it. Indeed, even Sun recommends basically > throwing more RAM at most problems. Again, tuning is often needed, and > I think it's fair to say that it can't be expected to work on arbitrary > workloads out of the box. > > Now, what about the other problems that have been reported in this > thread by Ivan and others? I don't think that it can be said that the > only problem that ZFS has is with memory. Unfortunately, it looks like > these "other" problems aren't well quantified, so I think that they are > being unfairly dismissed. But at the same time, maybe these other > problems are rare and unique enough that they represent very special > cases that won't be encountered by most people. But it also tells me > that ZFS is still immature, at least in FreeBSD. > > The universal need for tuning combined with the poorly understood > problem reports tells me that administrators considering ZFS should > expect to spend a fair amount of timing testing and tuning. Don't > expect it to work out of the box for your situation. That's not to > say that it's useless; there are certainly many people who can attest to > it working well for them. Just be prepared to spend time and possibly > money making it work, and be willing to provide good problem reports for > any non-memory related problems that you encounter. To be clear, in this thread I have been mostly restricting myself to discussion of kmem problems only, although I have also noted that there are known ZFS bugs including bugs that are unfixed even in solaris (the ZIL low memory deadlock is one of them). Indeed, pjd has a long list of bug reports from me :) I agree with the rest of this summary. Kris