Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 24 Nov 2016 11:42:41 -0700
From:      Alan Somers <asomers@freebsd.org>
To:        Rick Macklem <rmacklem@uoguelph.ca>
Cc:        Konstantin Belousov <kostikbel@gmail.com>, FreeBSD CURRENT <freebsd-current@freebsd.org>
Subject:   Re: NFSv4 performance degradation with 12.0-CURRENT client
Message-ID:  <CAOtMX2hBXAJN_udED-u5%2B6UznR2%2BW88xgb=RqKSZL65Z3%2BcKOw@mail.gmail.com>
In-Reply-To: <YTXPR01MB0189E0B1DB5B16EE6B388B7DDDB60@YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM>
References:  <CAOtMX2jJ2XoQyVG1c04QL7NTJn1pg38s=XEgecE38ea0QoFAOw@mail.gmail.com> <20161124090811.GO54029@kib.kiev.ua> <YTXPR01MB0189E0B1DB5B16EE6B388B7DDDB60@YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM>

next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Nov 24, 2016 at 5:53 AM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
>
> On Wed, Nov 23, 2016 at 10:17:25PM -0700, Alan Somers wrote:
>> I have a FreeBSD 10.3-RELEASE-p12 server exporting its home
>> directories over both NFSv3 and NFSv4.  I have a TrueOS client (based
>> on 12.0-CURRENT on the drm-next-4.7 branch, built on 28-October)
>> mounting the home directories over NFSv4.  At first, everything is
>> fine and performance is good.  But if the client does a buildworld
>> using sources on NFS and locally stored objects, performance slowly
>> degrades.  The degradation is most noticeable with metadata-heavy
>> operations.  For example, "ls -l" in a directory with 153 files takes
>> less than 0.1 seconds right after booting.  But the longer the
>> buildworld goes on, the slower it gets.  Eventually that same "ls -l"
>> takes 19 seconds.  When the home directories are mounted over NFSv3
>> instead, I see no degradation.
>>
>> top shows negligible CPU consumption on the server, and very high
>> consumption on the client when using NFSv4 (nearly 100%).  The
>> NFS-using process is spending almost all of its time in system mode,
>> and dtrace shows that almost all of its time is spent in
>> ncl_getpages().
>>
> A couple of things you could do when it slow (as well as what Kostik suggested):
> - nfsstat -c -e on client and nfsstat -e -s on server, to see what RPCs are being done
>   and how quickly. (nfsstat -s -e will also show you how big the DRC is, although a
>   large DRC should show up as increased CPU consumption on the server)
> - capture packets with tcpdump -s 0 -w test.pcap host <other-one>
>   - then you can email me test.pcap as an attachment. I can look at it in wireshark
>     and see if there seem to protocol and/or TCP issues. (You can look at in wireshark
>     yourself, the look for NFS4ERR_xxx, TCP segment retransmits...)
>
> If you are using either "intr" or "soft" on the mounts, try without those mount options.
> (The Bugs section of mount_nfs recommends against using them. If an RPC fails due to
>  these options, something called a seqid# can be "out of sync" between client/server and
>  that causes serious problems.)
> --> These seqid#s are not used by NFSv4.1, so you could try that by adding
>       "minorversion=1" to your mount options.
>
> Good luck with it, rick

I've reproduced the issue on stock FreeBSD 12, and I've also learned
that nullfs is a required factor.  Doing the buildworld directly on
the NFS mount doesn't cause any slowdown, but doing a buildworld on
the nullfs copy of the NFS mount does.  The slowdown affects the base
NFS mount as well as the nullfs copy.  Here is the nfsstat output for
both server and client duing "ls -al" on the client:

nfsstat -e -s -z

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
      800         0       121         0         0         2         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         8
    Mknod    Fsstat    Fsinfo  PathConf    Commit   LookupP   SetClId SetClIdCf
        0         0         0         0         1         3         0         0
     Open  OpenAttr OpenDwnGr  OpenCfrm DelePurge   DeleRet     GetFH      Lock
        0         0         0         0         0         0       123         0
    LockT     LockU     Close    Verify   NVerify     PutFH  PutPubFH PutRootFH
        0         0         0         0         0       674         0         0
    Renew RestoreFH    SaveFH   Secinfo RelLckOwn  V4Create
        0         0         0         0         0         0
Server:
Retfailed    Faults   Clients
        0         0         0
OpenOwner     Opens LockOwner     Locks    Delegs
        0         0         0         0         0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses CacheSize   TCPPeak
        0         0         0       674     16738     16738

nfsstat -e -c -z
Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
       60         0       119         0         0         0         0         0
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
        0         0         0         0         0         0         0         3
    Mknod    Fsstat    Fsinfo  PathConf    Commit   SetClId SetClIdCf      Lock
        0         0         0         0         0         0         0         0
    LockT     LockU      Open   OpenCfr
        0         0         0         0
OpenOwner     Opens LockOwner     Locks    Delegs  LocalOwn LocalOpen LocalLOwn
     5638    141453         0         0         0         0         0         0
LocalLock
        0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0       662
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
     1275        58       837       121         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses
        1         0         6         0         1         0

And here are the most popular stack traces of "ls -al", as observed by
dtrace.  The number beneath each stack is the number of times dtrace
observed that exact stack:

              kernel`bcmp+0x21
              kernel`vinactive+0xc6
              kernel`vputx+0x30e
              kernel`kern_statat+0x165
              kernel`sys_fstatat+0x2c
              kernel`amd64_syscall+0x314
              kernel`vputx+0x30e
              kernel`NDFREE+0xaa
              kernel`sys___acl_get_link+0x82
              kernel`amd64_syscall+0x314
              kernel`0xffffffff80eb95fb
               96

              kernel`nfscl_doclose+0x383
              kernel`vinactive+0xc6
              kernel`vputx+0x30e
              kernel`NDFREE+0xaa
              kernel`sys___acl_get_link+0x82
              kernel`amd64_syscall+0x314
              kernel`0xffffffff80eb95fb
              183

              kernel`nfscl_doclose+0x383
              kernel`vinactive+0xc6
              kernel`vputx+0x30e
              kernel`kern_statat+0x165
              kernel`sys_fstatat+0x2c
              kernel`amd64_syscall+0x314
              kernel`0xffffffff80eb95fb
              189

              kernel`lock_delay+0x52
              kernel`nfs_lookup+0x337
              kernel`VOP_LOOKUP_APV+0xda
              kernel`lookup+0x6a2
              kernel`namei+0x57e
              kernel`sys___acl_get_link+0x55
              kernel`amd64_syscall+0x314
              kernel`0xffffffff80eb95fb
              194

              kernel`lock_delay+0x52
              kernel`ncl_getattrcache+0x28
              kernel`nfs_getattr+0x92
              kernel`VOP_GETATTR_APV+0xda
              kernel`vn_stat+0xa3
              kernel`kern_statat+0xde
              kernel`sys_fstatat+0x2c
              kernel`amd64_syscall+0x314
              kernel`0xffffffff80eb95fb
              196

What role could nullfs be playing?



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2hBXAJN_udED-u5%2B6UznR2%2BW88xgb=RqKSZL65Z3%2BcKOw>