From owner-freebsd-stable@FreeBSD.ORG Wed Sep 11 13:08:35 2013 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id F2A6531C for ; Wed, 11 Sep 2013 13:08:34 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx12.netapp.com (mx12.netapp.com [216.240.18.77]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D49DC2AC7 for ; Wed, 11 Sep 2013 13:08:34 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.90,884,1371106800"; d="asc'?scan'208";a="88616596" Received: from vmwexceht02-prd.hq.netapp.com ([10.106.76.240]) by mx12-out.netapp.com with ESMTP; 11 Sep 2013 06:08:33 -0700 Received: from SACEXCMBX01-PRD.hq.netapp.com ([169.254.2.116]) by vmwexceht02-prd.hq.netapp.com ([10.106.76.240]) with mapi id 14.03.0123.003; Wed, 11 Sep 2013 06:08:33 -0700 From: "Eggert, Lars" To: Rick Macklem Subject: Re: nfsd CPU usage? Thread-Topic: nfsd CPU usage? Thread-Index: AQHOrto706tE4yigQ0+JIY2X/OKgRJnA4uQAgAAUxwA= Date: Wed, 11 Sep 2013 13:08:32 +0000 Message-ID: References: <995078453.21651811.1378900450650.JavaMail.root@uoguelph.ca> In-Reply-To: <995078453.21651811.1378900450650.JavaMail.root@uoguelph.ca> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [10.106.53.51] Content-Type: multipart/signed; boundary="Apple-Mail=_CF323C75-B1FD-4113-8CF8-E23280450295"; protocol="application/pgp-signature"; micalg=pgp-sha1 MIME-Version: 1.0 Cc: "" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Sep 2013 13:08:35 -0000 --Apple-Mail=_CF323C75-B1FD-4113-8CF8-E23280450295 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Thanks, I will watch out for the MFC and test. Lars On Sep 11, 2013, at 13:54, Rick Macklem wrote: > Lars Eggert wrote: >> Hi, >>=20 >> I'm seeing extremely high CPU usage withssh-st the new nfsd: >>=20 >> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 2280 root 102 0 9932K 1376K *nfs_c 0 320:11 100.00% >> nfsd{nfsd: service} >> 2280 root 102 0 9932K 1376K CPU7 7 319:47 100.00% >> nfsd{nfsd: service} >> 2280 root 102 0 9932K 1376K CPU5 5 318:25 100.00% >> nfsd{nfsd: service} >> 2280 root 102 0 9932K 1376K CPU6 6 318:20 100.00% >> nfsd{nfsd: service} >> 2280 root 52 0 9932K 1376K CPU0 0 317:32 100.00% >> nfsd{nfsd: service} >> 2280 root 102 0 9932K 1376K *nfs_c 1 315:41 99.17% >> nfsd{nfsd: service} >> 2280 root 52 0 9932K 1376K *nfs_c 4 320:22 98.78% >> nfsd{nfsd: master} >> 2280 root 102 0 9932K 1376K *nfs_c 1 317:10 98.10% >> nfsd{nfsd: service} >>=20 >> And this is at a few hundred KB/s with only a few clients: >>=20 >> ifstat -i igb1 10 >> igb1 >> KB/s in KB/s out >> 796.56 208.66 >> 431.19 232.36 >> 316.11 280.31 >> 1005.96 523.42 >> 1077.74 342.25 >> 340.63 217.73 >> 1067.96 330.56 >> 487.91 235.61 >>=20 >> Any ideas? >>=20 >> FreeBSD stanley.muccbc.hq.netapp.com 9.2-PRERELEASE FreeBSD >> 9.2-PRERELEASE #7: Wed Sep 4 11:06:31 CEST 2013 >> root@stanley.muccbc.hq.netapp.com:/usr/obj/usr/src/sys/STANLEY >> amd64 >>=20 >> Thanks, >> Lars >>=20 > There is a patch in head (r254337) that I believe handles this. > It will be MFC'd to stable/9 in about a week, unless someone finds > problems with it before then. > If you want a semantically equivalent (but uglier code) patch, > you can find it here: > http://people.freebsd.org/~rmacklem/drc4-stable9.patch > After applying the patch, you need to set sysctl variable(s), > to avoid the aggressive trimming of stale DRC entries. Garrett > Wollman suggests the following for a large server: > vfs.nfsd.tcphighwater=3D100000 > vfs.nfsd.tcpcachetimeout=3D300 (5 minutes instead of default of = several hrs) >=20 > You can also use the sysctl > vfs.nfsd.cachetcp=3D0 > to disable use of the DRC for TCP. >=20 > The old nfs server did not use the DRC for TCP. The assumption being = that > TCP layer retransmits are good enough to maintain reliable RPC = transport. > Unfortuantely, you can get file corruption when the server reboots or > there is a network partitioning, if the client chooses to redo the RPC > over TCP (clients always do this after having to create a new TCP = connection). > In other words, vfs.nfsd.cachetcp=3D0 is roughly what the old nfsd = did. >=20 > If you don't want to patch the 9.2 code, you can edit the sources > (sys/fs/nfsserver/nfs_nfsdcache.c) and change the line: > static int nfsrc_tcpnonidempotent =3D 1; > to > static int nfsrc_tcpnonidempotent =3D 0; > to do the same thing as vfs.nfsd.cachetcp=3D0 >=20 > rick --Apple-Mail=_CF323C75-B1FD-4113-8CF8-E23280450295 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQCVAwUBUjBrUNZcnpRveo1xAQJhNgQAhKrslgI4FseWob/uGeX9BBLHZkPEE2Tn mpz1wnIwiW8b/00ugi1R+mhJ2ZNxR8Srs8BYAvBCutbvOGrjXP8pGDsawyKZ2HpK zmDvDHj42guTEFMcHP31/uorZNJAbW3gHnMozd3ZhX0T90rGN/tEfde5rQNgoEGA pirG56xck2o= =KVli -----END PGP SIGNATURE----- --Apple-Mail=_CF323C75-B1FD-4113-8CF8-E23280450295--