From owner-freebsd-fs@FreeBSD.ORG Mon Sep 9 22:58:06 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 58DDECE3 for ; Mon, 9 Sep 2013 22:58:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 2125629D8 for ; Mon, 9 Sep 2013 22:58:05 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqUEAJJRLlKDaFve/2dsb2JhbABYAxaDKUsGgyq+dIE7dIIlAQEBAwEBAQEgKyALBRYOCgICDRkCKQEJJgYIBwQBHASHWwYHBbEpkgmBKY0JgQUkEAcRgliBNAOVLIN4iwuFLIM8IDKBAzk X-IronPort-AV: E=Sophos;i="4.90,874,1371096000"; d="scan'208";a="50744805" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 09 Sep 2013 18:57:59 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 06311B3F2E; Mon, 9 Sep 2013 18:57:59 -0400 (EDT) Date: Mon, 9 Sep 2013 18:57:59 -0400 (EDT) From: Rick Macklem To: Attila Nagy Message-ID: <1721695444.20803113.1378767479011.JavaMail.root@uoguelph.ca> In-Reply-To: <522E4AC5.4040606@fsn.hu> Subject: Re: High CPU usage with newnfs(d) - seems to be a cache issue MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Sep 2013 22:58:06 -0000 Attila Nagy wrote: > Hi, > > I've observed some insane CPU usage on stable/9@r255367. > About the machine: > CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2400.14-MHz > K8-class CPU) > real memory = 34359738368 (32768 MB) > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads > > It does some NFS serving like this (now running oldnfs) -not quite > peak > times actually: > # nfsstat -w 1 -os > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 763 7206 1 175 92 0 915 3589 > 748 7665 10 131 60 0 905 2923 > 787 9657 23 204 50 0 974 2387 > 517 9881 9 150 41 0 572 2321 > 709 8708 71 235 70 0 1220 3271 > 621 9157 9 254 208 0 928 2563 > 699 5336 29 271 103 0 1242 3448 > 656 4291 11 201 209 0 1119 3908 > 506 3722 0 215 183 0 970 2516 > 698 1476 1 151 66 0 903 2094 > 501 2865 11 268 117 0 995 1392 > 638 6284 46 233 47 0 1096 4847 > 893 7909 47 175 73 0 870 4070 > 651 3936 48 255 51 0 955 2514 > 424 4211 17 223 29 0 745 1458 > 589 8197 26 199 39 0 918 2983 > > It's being hammered by about 40 machines on multiple connections (it > has > 35 UFS file systems exported). > > When running newnfs (admittedly in some stupid way, with -n 32, the > profiling was made with this, maybe this causes some lock > contention), > it occasionally eats 1600% CPU (means: 0 idle). > Lowering the thread number doesn't really solves the problem, I've > seen > -n X*100 CPU usage peaks lately on machines with lower (4-8) -n > counts... > > Doing a profiling with pmc shows that most of the time is spent in > nfsrvd_updatecache and nfsrvd_getcache: > http://pastebin.com/knyppv4d > > Switching back to oldnfsd (even with -n 32) gives a stable 50-60% CPU > usage (out of the "possible" 1600%) when loaded. > > I know that there are some changes regarding this cache in the > CURRENT > code (along with the possibility to set some values with sysctls), > but I > can't run CURRENT. > > Any ideas on how to improve newnfsd, so we can continue serving NFS > in > the future days, where I won't be able to switch back to the old one? > :) > Well, I put a 1 month MFC on r254337 (which I believe fixes this), so it should be in stable/9 in about a week. Alternately, an uglier (but semantically equivalent) patch can be found at: http://people.freebsd.org/~rmacklem/drc4-stable9.patch rick > Thanks, > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >