Date: Wed, 11 Sep 2013 07:54:10 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Lars Eggert <lars@netapp.com> Cc: stable@freebsd.org Subject: Re: nfsd CPU usage? Message-ID: <995078453.21651811.1378900450650.JavaMail.root@uoguelph.ca> In-Reply-To: <404C967C-4BFB-424C-A5E0-6ACE6576249C@netapp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Lars Eggert wrote:
> Hi,
>
> I'm seeing extremely high CPU usage withssh-st the new nfsd:
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU
> COMMAND
> 2280 root 102 0 9932K 1376K *nfs_c 0 320:11 100.00%
> nfsd{nfsd: service}
> 2280 root 102 0 9932K 1376K CPU7 7 319:47 100.00%
> nfsd{nfsd: service}
> 2280 root 102 0 9932K 1376K CPU5 5 318:25 100.00%
> nfsd{nfsd: service}
> 2280 root 102 0 9932K 1376K CPU6 6 318:20 100.00%
> nfsd{nfsd: service}
> 2280 root 52 0 9932K 1376K CPU0 0 317:32 100.00%
> nfsd{nfsd: service}
> 2280 root 102 0 9932K 1376K *nfs_c 1 315:41 99.17%
> nfsd{nfsd: service}
> 2280 root 52 0 9932K 1376K *nfs_c 4 320:22 98.78%
> nfsd{nfsd: master}
> 2280 root 102 0 9932K 1376K *nfs_c 1 317:10 98.10%
> nfsd{nfsd: service}
>
> And this is at a few hundred KB/s with only a few clients:
>
> ifstat -i igb1 10
> igb1
> KB/s in KB/s out
> 796.56 208.66
> 431.19 232.36
> 316.11 280.31
> 1005.96 523.42
> 1077.74 342.25
> 340.63 217.73
> 1067.96 330.56
> 487.91 235.61
>
> Any ideas?
>
> FreeBSD stanley.muccbc.hq.netapp.com 9.2-PRERELEASE FreeBSD
> 9.2-PRERELEASE #7: Wed Sep 4 11:06:31 CEST 2013
> root@stanley.muccbc.hq.netapp.com:/usr/obj/usr/src/sys/STANLEY
> amd64
>
> Thanks,
> Lars
>
There is a patch in head (r254337) that I believe handles this.
It will be MFC'd to stable/9 in about a week, unless someone finds
problems with it before then.
If you want a semantically equivalent (but uglier code) patch,
you can find it here:
http://people.freebsd.org/~rmacklem/drc4-stable9.patch
After applying the patch, you need to set sysctl variable(s),
to avoid the aggressive trimming of stale DRC entries. Garrett
Wollman suggests the following for a large server:
vfs.nfsd.tcphighwater=100000
vfs.nfsd.tcpcachetimeout=300 (5 minutes instead of default of several hrs)
You can also use the sysctl
vfs.nfsd.cachetcp=0
to disable use of the DRC for TCP.
The old nfs server did not use the DRC for TCP. The assumption being that
TCP layer retransmits are good enough to maintain reliable RPC transport.
Unfortuantely, you can get file corruption when the server reboots or
there is a network partitioning, if the client chooses to redo the RPC
over TCP (clients always do this after having to create a new TCP connection).
In other words, vfs.nfsd.cachetcp=0 is roughly what the old nfsd did.
If you don't want to patch the 9.2 code, you can edit the sources
(sys/fs/nfsserver/nfs_nfsdcache.c) and change the line:
static int nfsrc_tcpnonidempotent = 1;
to
static int nfsrc_tcpnonidempotent = 0;
to do the same thing as vfs.nfsd.cachetcp=0
rick
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?995078453.21651811.1378900450650.JavaMail.root>