From owner-freebsd-current@FreeBSD.ORG Mon Dec 8 03:24:42 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45F091065672 for ; Mon, 8 Dec 2008 03:24:42 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mx.egr.msu.edu (surfnturf.egr.msu.edu [35.9.37.164]) by mx1.freebsd.org (Postfix) with ESMTP id 16E0C8FC13 for ; Mon, 8 Dec 2008 03:24:41 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from localhost (localhost [127.0.0.1]) by mx.egr.msu.edu (Postfix) with ESMTP id 37C7C71EE75; Sun, 7 Dec 2008 22:24:41 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mx.egr.msu.edu ([127.0.0.1]) by localhost (surfnturf.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LyfrkqVq+amQ; Sun, 7 Dec 2008 22:24:41 -0500 (EST) Received: from localhost (daemon.egr.msu.edu [35.9.44.65]) by mx.egr.msu.edu (Postfix) with ESMTP id E8C9B71EE72; Sun, 7 Dec 2008 22:24:40 -0500 (EST) Received: by localhost (Postfix, from userid 21281) id CFFA5A6A; Sun, 7 Dec 2008 22:24:40 -0500 (EST) Date: Sun, 7 Dec 2008 22:24:40 -0500 From: Adam McDougall To: Stefan Bethke Message-ID: <20081208032440.GU93166@egr.msu.edu> References: <20081117205526.GC1733@garage.freebsd.pl> <20081202203308.GA13818@hyperion.scode.org> <200812021254.21242.fjwcash@gmail.com> <20081202232924.GA19134@hyperion.scode.org> <31C70CBC-488A-4A9A-A642-37855E8F1DD1@lassitu.de> <5C97E586-4296-4835-A103-FD273B2D7A4F@lassitu.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5C97E586-4296-4835-A103-FD273B2D7A4F@lassitu.de> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: FreeBSD Current Subject: Re: ZFS gets stuck X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Dec 2008 03:24:42 -0000 On Sun, Dec 07, 2008 at 11:21:13AM +0100, Stefan Bethke wrote: For the past week, I've been stress testing two new boxes by running make -j4 universe. /usr/src is on ufs, /usr/obj on zfs, backed by a single disk pool. Every so often (about once every one or two days), processes start getting wedged on accessing the zfs file systems. FreeBSD lokschuppen.lassitu.de 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Wed Dec 3 07:05:03 UTC 2008 root@lokschuppen.lassitu.de:/usr/obj/usr/ src/sys/EISENBOOT amd64 So far, I've had this in loader.conf: vfs.zfs.arc_max="512M" vfs.zfs.prefetch_disable="1" I'm now adding vfs.zfs.zil_disable="1" to see if that makes a difference. Is there anything in particular people would want me to check out? Kernel is GENERIC minus a number of devices, and without INVARIANTS and WITNESS. I have one system running a recent -current with 2G of ram that would tend to get stuck almost nightly during a nightly rsync, and have been trying to tune it to get rid of the problems. I've made it about 6 days so far using just: vm.kmem_size=2G vm.kmem_size_max=2G vfs.zfs.arc_max=512M With these settings, I had two rsyncs running constantly for probably 12 hours on the first day or two and still once nightly, so far so good. I tried leaving the kmem settings out and while vm.kmem_size_max appeared to auto tune to approx 4.5G, I still had a out of kmem panic (a low number around 600 or 900M, I don't remember) until I boosted both kmem settings to 2G. I previously had ~1.6G for the kmem settings from when that was the highest they could be set, and maybe that wasn't high enough with recent -current because it had hangs every few days as opposed to every few months with zfs "6" (same zfs as in -stable). I might just be having a string of good luck, and the data that rsync transfers might have played a role (it changes a varying amount each day). If it looks fairly stable then I will apply the settings to a different, busier server with ZFS.