From owner-freebsd-fs@freebsd.org Sun Feb 14 08:41:34 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A44A9AA1D15 for ; Sun, 14 Feb 2016 08:41:34 +0000 (UTC) (envelope-from zanchey@ucc.gu.uwa.edu.au) Received: from mail-ext-sout1.uwa.edu.au (mail-ext-sout1.uwa.edu.au [130.95.128.72]) by mx1.freebsd.org (Postfix) with ESMTP id 2B2DD1B0C for ; Sun, 14 Feb 2016 08:41:32 +0000 (UTC) (envelope-from zanchey@ucc.gu.uwa.edu.au) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DIBQC+PMBW/8+AX4JUCoQMbaZGAQEBAQEBBpU5IYI8gzACgW4BAQEBAQGBC4RCAQEEOj8QCw4KLiwrBhOIGg62eIQKAQEBAQYBAQEBARcEhUmCO4JChAgFhF8FjhqIX4VPiWKHaYUvjj5iggIZgVVdAYhOgTgBAQE X-IPAS-Result: A2DIBQC+PMBW/8+AX4JUCoQMbaZGAQEBAQEBBpU5IYI8gzACgW4BAQEBAQGBC4RCAQEEOj8QCw4KLiwrBhOIGg62eIQKAQEBAQYBAQEBARcEhUmCO4JChAgFhF8FjhqIX4VPiWKHaYUvjj5iggIZgVVdAYhOgTgBAQE X-IronPort-AV: E=Sophos;i="5.22,444,1449504000"; d="scan'208";a="199840790" Received: from f5-new.net.uwa.edu.au (HELO mooneye.ucc.gu.uwa.edu.au) ([130.95.128.207]) by mail-ext-out1.uwa.edu.au with SMTP; 14 Feb 2016 16:40:22 +0800 Received: by mooneye.ucc.gu.uwa.edu.au (Postfix, from userid 801) id DA8353C04E; Sun, 14 Feb 2016 16:40:22 +0800 (AWST) Received: from motsugo.ucc.gu.uwa.edu.au (motsugo.ucc.gu.uwa.edu.au [130.95.13.7]) by mooneye.ucc.gu.uwa.edu.au (Postfix) with ESMTP id AB8A93C04E; Sun, 14 Feb 2016 16:40:22 +0800 (AWST) Received: by motsugo.ucc.gu.uwa.edu.au (Postfix, from userid 11251) id A566420083; Sun, 14 Feb 2016 16:40:22 +0800 (AWST) Received: from localhost (localhost [127.0.0.1]) by motsugo.ucc.gu.uwa.edu.au (Postfix) with ESMTP id A190220065; Sun, 14 Feb 2016 16:40:22 +0800 (AWST) Date: Sun, 14 Feb 2016 16:40:22 +0800 (AWST) From: David Adam To: Tom Curry cc: FreeBSD Filesystems Subject: Re: Poor ZFS+NFSv3 read/write performance and panic In-Reply-To: Message-ID: References: User-Agent: Alpine 2.11 (DEB 23 2013-08-11) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Feb 2016 08:41:34 -0000 On Mon, 8 Feb 2016, Tom Curry wrote: > On Sun, Feb 7, 2016 at 11:58 AM, David Adam > wrote: > > > Just wondering if anyone has any idea how to identify which devices are > > implicated in ZFS' vdev_deadman(). I have updated the firmware on the > > mps(4) card that has our disks attached but that hasn't helped. > > I too ran into this problem and spent quite some time troubleshooting > hardware. For me it turns out it was not hardware at all, but software. > Specifically the ZFS ARC. Looking at your stack I see some arc reclaim up > top, it's possible you're running into the same issue. There is a monster > of a PR that details this here > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 > > If you would like to test this theory out, the fastest way is to limit the > ARC by adding the following to /boot/loader.conf and rebooting > vfs.zfs.arc_max="24G" > > Replacing 24G with what makes sense for your system, aim for 3/4 of total > memory for starters. If this solves the problem there are more scientific > methods to a permanent fix, one would be applying the patch in the PR > above, another would be a more finely tuned arc_max value. Thanks Tom - this certainly did sound promising, but setting the ARC to 11G of our 16G of RAM didn't help. `zfs-stats` confirmed that the ARC was the expected size and that there was still 461 MB of RAM free. We'll keep looking! David Adam zanchey@ucc.gu.uwa.edu.au