Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 11 Sep 2013 13:08:32 +0000
From:      "Eggert, Lars" <lars@netapp.com>
To:        Rick Macklem <rmacklem@uoguelph.ca>
Cc:        "<stable@freebsd.org>" <stable@freebsd.org>
Subject:   Re: nfsd CPU usage?
Message-ID:  <A24A8E32-53AA-45F1-ADDA-2C55863018B5@netapp.com>
In-Reply-To: <995078453.21651811.1378900450650.JavaMail.root@uoguelph.ca>
References:  <995078453.21651811.1378900450650.JavaMail.root@uoguelph.ca>

next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_CF323C75-B1FD-4113-8CF8-E23280450295
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Thanks, I will watch out for the MFC and test.

Lars

On Sep 11, 2013, at 13:54, Rick Macklem <rmacklem@uoguelph.ca> wrote:

> Lars Eggert wrote:
>> Hi,
>>=20
>> I'm seeing extremely high CPU usage withssh-st the new nfsd:
>>=20
>>  PID USERNAME   PRI NICE   SIZE    RES STATE   C   TIME   WCPU
>>  COMMAND
>> 2280 root       102    0  9932K  1376K *nfs_c  0 320:11 100.00%
>> nfsd{nfsd: service}
>> 2280 root       102    0  9932K  1376K CPU7    7 319:47 100.00%
>> nfsd{nfsd: service}
>> 2280 root       102    0  9932K  1376K CPU5    5 318:25 100.00%
>> nfsd{nfsd: service}
>> 2280 root       102    0  9932K  1376K CPU6    6 318:20 100.00%
>> nfsd{nfsd: service}
>> 2280 root        52    0  9932K  1376K CPU0    0 317:32 100.00%
>> nfsd{nfsd: service}
>> 2280 root       102    0  9932K  1376K *nfs_c  1 315:41 99.17%
>> nfsd{nfsd: service}
>> 2280 root        52    0  9932K  1376K *nfs_c  4 320:22 98.78%
>> nfsd{nfsd: master}
>> 2280 root       102    0  9932K  1376K *nfs_c  1 317:10 98.10%
>> nfsd{nfsd: service}
>>=20
>> And this is at a few hundred KB/s with only a few clients:
>>=20
>> ifstat -i igb1 10
>>       igb1
>> KB/s in  KB/s out
>>  796.56    208.66
>>  431.19    232.36
>>  316.11    280.31
>> 1005.96    523.42
>> 1077.74    342.25
>>  340.63    217.73
>> 1067.96    330.56
>>  487.91    235.61
>>=20
>> Any ideas?
>>=20
>> FreeBSD stanley.muccbc.hq.netapp.com 9.2-PRERELEASE FreeBSD
>> 9.2-PRERELEASE #7: Wed Sep  4 11:06:31 CEST 2013
>>    root@stanley.muccbc.hq.netapp.com:/usr/obj/usr/src/sys/STANLEY
>> amd64
>>=20
>> Thanks,
>> Lars
>>=20
> There is a patch in head (r254337) that I believe handles this.
> It will be MFC'd to stable/9 in about a week, unless someone finds
> problems with it before then.
> If you want a semantically equivalent (but uglier code) patch,
> you can find it here:
>  http://people.freebsd.org/~rmacklem/drc4-stable9.patch
> After applying the patch, you need to set sysctl variable(s),
> to avoid the aggressive trimming of stale DRC entries. Garrett
> Wollman suggests the following for a large server:
> vfs.nfsd.tcphighwater=3D100000
> vfs.nfsd.tcpcachetimeout=3D300 (5 minutes instead of default of =
several hrs)
>=20
> You can also use the sysctl
> vfs.nfsd.cachetcp=3D0
> to disable use of the DRC for TCP.
>=20
> The old nfs server did not use the DRC for TCP. The assumption being =
that
> TCP layer retransmits are good enough to maintain reliable RPC =
transport.
> Unfortuantely, you can get file corruption when the server reboots or
> there is a network partitioning, if the client chooses to redo the RPC
> over TCP (clients always do this after having to create a new TCP =
connection).
> In other words, vfs.nfsd.cachetcp=3D0 is roughly what the old nfsd =
did.
>=20
> If you don't want to patch the 9.2 code, you can edit the sources
> (sys/fs/nfsserver/nfs_nfsdcache.c) and change the line:
> static int nfsrc_tcpnonidempotent =3D 1;
> to
> static int nfsrc_tcpnonidempotent =3D 0;
> to do the same thing as vfs.nfsd.cachetcp=3D0
>=20
> rick


--Apple-Mail=_CF323C75-B1FD-4113-8CF8-E23280450295
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="signature.asc"
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org

iQCVAwUBUjBrUNZcnpRveo1xAQJhNgQAhKrslgI4FseWob/uGeX9BBLHZkPEE2Tn
mpz1wnIwiW8b/00ugi1R+mhJ2ZNxR8Srs8BYAvBCutbvOGrjXP8pGDsawyKZ2HpK
zmDvDHj42guTEFMcHP31/uorZNJAbW3gHnMozd3ZhX0T90rGN/tEfde5rQNgoEGA
pirG56xck2o=
=KVli
-----END PGP SIGNATURE-----

--Apple-Mail=_CF323C75-B1FD-4113-8CF8-E23280450295--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A24A8E32-53AA-45F1-ADDA-2C55863018B5>