From owner-freebsd-fs@FreeBSD.ORG Thu Aug 18 12:58:53 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91A21106566B for ; Thu, 18 Aug 2011 12:58:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 2D3C68FC08 for ; Thu, 18 Aug 2011 12:58:52 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p7ICwnag045817 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 18 Aug 2011 15:58:49 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p7ICwnfA011247; Thu, 18 Aug 2011 15:58:49 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p7ICwnFt011246; Thu, 18 Aug 2011 15:58:49 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 18 Aug 2011 15:58:49 +0300 From: Kostik Belousov To: Rick Macklem Message-ID: <20110818125849.GE17489@deviant.kiev.zoral.com.ua> References: <20110817135230.GW17489@deviant.kiev.zoral.com.ua> <1632122286.297610.1313619533702.JavaMail.root@erie.cs.uoguelph.ca> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ADZ8S6Yea/b683e6" Content-Disposition: inline In-Reply-To: <1632122286.297610.1313619533702.JavaMail.root@erie.cs.uoguelph.ca> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, onwahe@gmail.com Subject: Re: NFS calculation of max commit size X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Aug 2011 12:58:53 -0000 --ADZ8S6Yea/b683e6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 17, 2011 at 06:18:53PM -0400, Rick Macklem wrote: > Kostik Belousov wrote: > > On Wed, Aug 17, 2011 at 09:15:15AM -0400, Rick Macklem wrote: > > > > > > I think that any fraction of hibufspace should be sufficient to > > > avoid > > > the deadlock. Also, since the buffer cache code doesn't use vnode > > > locking > > > these days, I'm not even sure if write backs are blocked by the > > > wrire > > > vnode op in progress. (ie. I'm not sure the deadlock it originally > > > fixed > > > would still happen without it.) > >=20 > > bufdaemon definitely acquires vnode lock when flushing dirty buffer, > > this was a problem on its own. I think you refer to the nfsiod > > operation. > >=20 > Ok, so I think this means that the deadlock can still occur. > I haven't yet played with the code, but I now think I might unedrstand > the logic behind dividing by "(desiredvnodes / 1000)". >=20 > If a single large write is happening to one NFS vnode, setting > nm_wcommitsize to any fraction of hibufspace should avoid the deadlock, > I think. (If I understand it correctly, the deadlock occurs when an > NFS VOP_WRITE() runs out of buffer cache and no buffer cache blocks > can be cleaned out because it is holding a lock on the vnode.) No, if nfs write vop tries to allocate a new buffer, then vfs_bio.c will note that attempt is made to allocate with the vnode lock held, and will do a pass of the dirty cache flushing buffers owned by the vnode. See a call to buf_do_flush() from getnewbuf() and buf_do_flush() code itself. This is what I referred to as 'a problem on its own'. The change helped to fix a bufdaemon deadlock you described, that indeed happen relatively often. >=20 > But, what happens if K processes concurrently do large writes on K > NFS vnodes? > - It seems to me that they call could deadlock when the buffer cache > becomes exhausted, since they all hold locks on their respective > vnodes and, therefore, none of the dirty buffers can be flushed. > - If this is correct, then I think the only "safe" answer is: > nm_wcommitsize =3D hibufspace / desiredvnodes; > since it is possible that almost all vnodes could be assigned to > NFS files being written concurrently with large writes. > However, this would result in an absurdly low value for nm_wcommitsize. >=20 > --> My best guess is the original author assumed that 0.1% of all vnodes > would be a reasonable upper bound on the number being written by NFS > concurrently with large writes. >=20 > By the way, since nm_wcommitsize is applied to a single write, it only > affects a single write(2) syscall of more than nm_wcommitsize bytes of > data. (The PR refers to a writev() of 60Mbytes in size.) > I honestly have no idea how many apps. do write() syscalls of megabytes > in size, so I'm not sure how important it would be to make it larger > than "hibufspace / (desiredvnodes / 1000)", which is about 2Mbytes on > the 256Mbyte laptop I have here without any tuning tweaks? >=20 > I think there might be a better way to do this than calculating a > fixed "guestimate" for nm_wcommitsize and then using it for the life > of the NFS mount. > - The NFS VOP_WRITE() can keep track of a running total of how many > bytes is being written: > - add uio_resid to this running total at the beginning of the VOP_WRITE= () > and subtract it back out at the end of VOP_WRITE(). > - if this running total exceeds something like 80% of hibufspace, then > do synchronous writes (ie. use that test instead of > if (nm_wcommitsize < uio->uio_resid) to make the decision. >=20 > Does this sound reasonable to others? > (This is actually getting interesting. Who would have guessed that a > divide by zero bug report would lead to this...) >=20 > rick > > There is another op that is performed without holding the vnode lock > > consistently from (old)nfs code, namely, truncation. It would be > > useful > > to fix this. Please see r188386. --ADZ8S6Yea/b683e6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk5NDIgACgkQC3+MBN1Mb4goIwCgsDM23cix0FchRJmbDXilSyZY JEkAoJ6o/edVJVLaeF50bY2E88rTPoWR =MwS4 -----END PGP SIGNATURE----- --ADZ8S6Yea/b683e6--