Date: Wed, 7 Sep 2011 16:17:54 -0700 From: David Brodbeck <brodbd@uw.edu> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: freebsd-fs@freebsd.org Subject: Re: Slow NFSv4 performance, was: Re: ZFSv28+NFSv4 poor file creation performance, "sync=disabled" has no effect Message-ID: <CAHHaOubiQTXNvWUHUV_fyOk83bTyQQ=hUi_fX_mTD_FSCZPuvg@mail.gmail.com> In-Reply-To: <1356496286.948283.1315433975522.JavaMail.root@erie.cs.uoguelph.ca> References: <CAHHaOuYda8S09E89jhWV4mE6r8_gnQhr_jn%2BDqM%2B0PeNnUSpqw@mail.gmail.com> <1356496286.948283.1315433975522.JavaMail.root@erie.cs.uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Sep 7, 2011 at 3:19 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote: > Yea, I just tried a quick test using Fedora15 mounting the FreeBSD NFS > server and then extracted a small tarball. Here's what the RPC counts > were for NFSv3 and NFSv4 mounts (Open replaces Create for file creation > in NFSv4): > > Client Getattr Setattr Read Write Create Open > Fedora-NFSv3 12 249 35 75 60 0 > Fedora-NFSv4 744 249 35 75 0 121 > FreeBSDNFSv3 64 189 35 75 60 0 > FreeBSDNFSv4 943 189 35 75 0 61 > > Two obvious things: > 1 - Fedora15 does 2 Opens for each create, for NFSv4. > 2 - NFSv4 does a massive # of Getattrs, however most of these will > be in the compound RPCs with other Ops, so it doesn't mean > that these are creating a larger load on the server. > Put another way, the Getattrs done as a part of other RPCs > don't get counted for NFSv3, but they are separate Ops in the > NFSv4 compound and, therefore, do get counted. > *** This was NOT a serious benchmark and I didn't bother to list all > the RPC counts. Do not read this as a serious performance comparison, > it was simply meant to get a hint as to why there was a difference > between NFSv3 and NFSv4 for this case. > I've also found this isn't just a Linux client problem. I tested a FreeBSD-9.0 client against a FreeBSD-9.0 server. Bonnie++ won't run on FreeBSD NFS mounts due to PR kern/26142, so I hacked up a quick script that times how long it takes to create 10,000 files. I averaged times over four runs; the backing filesystem was a UFS ramdisk. FreeBSD client -> FreeBSD server, NFSv4: 95.43s FreeBSD client -> FreeBSD server, NFSv3: 24.77s So basically NFSv4 against a FreeBSD server is nearly 4x slower at doing file creation than NFSv3, even when there's no disk latency involved. I did the same test with Linux and the results were similar, although the performance gap wasn't as wide: Linux client -> FreeBSD server, NFSv4: 71.25s Linux client -> FreeBSD server, NFSv3: 34.08s Sorry, but I can't think of anything within the NFS server. It just does > the operations that the client hands to it by performing the VOP_xxx() > calls. > I will take a look at a packet trace, but I doubt I'll learn anything > useful > from it. > Hm, okay. Unfortunately this has a big impact on us, because file creation speed happens to be one of the more user-visible aspects of server performance on our system. -- David Brodbeck System Administrator, Linguistics University of Washington
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHHaOubiQTXNvWUHUV_fyOk83bTyQQ=hUi_fX_mTD_FSCZPuvg>