From owner-freebsd-fs@FreeBSD.ORG Thu Apr 2 15:51:11 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3A8B0EE2 for ; Thu, 2 Apr 2015 15:51:11 +0000 (UTC) Received: from smtp102-5.vfemail.net (eightfive.vfemail.net [96.30.253.185]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EB3598CC for ; Thu, 2 Apr 2015 15:51:10 +0000 (UTC) Received: (qmail 19216 invoked by uid 89); 2 Apr 2015 15:51:06 -0000 Received: by simscan 1.4.0 ppid: 19159, pid: 19212, t: 0.0815s scanners:none Received: from unknown (HELO d3d3MTExQDE0Mjc5ODk4NjY=) (cmlja0BoYXZva21vbi5jb21AMTQyNzk4OTg2Ng==@MTcyLjE2LjEwMC45M0AxNDI3OTg5ODY2) by 172.16.100.62 with ESMTPA; 2 Apr 2015 15:51:06 -0000 Date: Thu, 02 Apr 2015 10:50:40 -0500 Message-ID: <20150402105040.Horde.DpcVnMHXCV_MvaXmGcnU1g8@www.vfemail.net> From: Rick Romero To: Adam Guimont Subject: Re: NFSD high CPU usage References: <20150401154314.Horde.e_w-9XEJOaa4SwYyNLlttA3@www.vfemail.net> <551D5CBC.1010009@tezzaron.com> In-Reply-To: <551D5CBC.1010009@tezzaron.com> User-Agent: Internet Messaging Program (IMP) H5 (6.2.2) X-VFEmail-Originating-IP: MTIuMzEuMTAwLjE0Ng== X-VFEmail-AntiSpam: Notify admin@vfemail.net of any spam, and include VFEmail headers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes Content-Transfer-Encoding: 8bit Content-Disposition: inline Content-Description: Plaintext Message X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2015 15:51:11 -0000 Quoting Adam Guimont : > Rick Romero wrote: >> Does your ZFS pool have log devices? >> How does gstat -d   look? >> >> If the drives are busy, try adding >> vfs.nfsd.async: 0 > > No log devices but the disks are not busy when this happens. > > I have an atop snapshot from the last time it happened: > http://pastebin.com/raw.php?i=LQjbKTXR Are the disks busy before it happens?   I'm far from an expert, but when running ZFS with NFS, I've had a lot of issues.  My final resolutions were to turn ASYNC off and have log devices and I even have SSD volumes now. Otherwise under load the NFS server gets hung up. It never seemed to happen on UFS, but due to the number of small files I have, ZFS provides the best backup functionality. I'm now trying to move all functions from NFS (to more TCP client/server). You have different info than I've gathered, and it might be because of usage. I actively use the system that I've seen NFS dump on, so I see the slowness beginning. Once NFS dies, the drive load goes back to normal. I wonder, if maybe you are just managing a system for others, and you don't see it until after the fact?  Just a thought based on my limited experience. Rick