Date: Wed, 7 Sep 2011 18:19:35 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: David Brodbeck <brodbd@uw.edu> Cc: freebsd-fs@freebsd.org Subject: Re: Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect Message-ID: <1356496286.948283.1315433975522.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <CAHHaOuYda8S09E89jhWV4mE6r8_gnQhr_jn%2BDqM%2B0PeNnUSpqw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
David Brodbeck wrote: > On Mon, Sep 5, 2011 at 2:16 AM, Pawel Jakub Dawidek <pjd@freebsd.org> > wrote: > > > It would be good if you could try your test on FreeBSD directly. > > This > > way we could see if ZFS is to blame or NFS. > > > > Okay, it appears it is in fact an NFSv4 problem, and I've been barking > up > the wrong tree. I shared out an NFSv4 mountpoint with a UFS ramdisk as > the > backing store, and I got the same slow results. I was fooled by the > fact > that the numbers were similar to what I got using OpenSolaris with the > ZIL > enabled. (This is what I get for making assumptions.) > > Interestingly enough, if I use NFSv3 instead of NFSv4, performance > increases > dramatically. > Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS server and then extracted a small tarball. Here's what the RPC counts were for NFSv3 and NFSv4 mounts (Open replaces Create for file creation in NFSv4): Client Getattr Setattr Read Write Create Open Fedora-NFSv3 12 249 35 75 60 0 Fedora-NFSv4 744 249 35 75 0 121 FreeBSDNFSv3 64 189 35 75 60 0 FreeBSDNFSv4 943 189 35 75 0 61 Two obvious things: 1 - Fedora15 does 2 Opens for each create, for NFSv4. 2 - NFSv4 does a massive # of Getattrs, however most of these will be in the compound RPCs with other Ops, so it doesn't mean that these are creating a larger load on the server. Put another way, the Getattrs done as a part of other RPCs don't get counted for NFSv3, but they are separate Ops in the NFSv4 compound and, therefore, do get counted. *** This was NOT a serious benchmark and I didn't bother to list all the RPC counts. Do not read this as a serious performance comparison, it was simply meant to get a hint as to why there was a difference between NFSv3 and NFSv4 for this case. So, my best guess is that doing twice as many Opens (vs NFSv3 Create RPCs) is why you see the difference. I'd also guess that disabling the ZIL on ZFS made the Opens perform faster, which helped alleviate the problem. (An NFSv4 Open operation will end up doing both VOP_GETATTR() and VOP_CREATE(). It will also do a VOP_SETATTR() for the exclusive open case.) > It appears it's FreeBSD's NFSv4 server, or the way it's interacting > with > Linux's client, that's the culprit here. I'm not sure if there are > knobs I > should be tweaking to make it perform better; any suggestions on what > to try > next? > Sorry, but I can't think of anything within the NFS server. It just does the operations that the client hands to it by performing the VOP_xxx() calls. I will take a look at a packet trace, but I doubt I'll learn anything useful from it. rick > -- > David Brodbeck > System Administrator, Linguistics > University of Washington > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1356496286.948283.1315433975522.JavaMail.root>