Date: Mon, 9 Sep 2013 18:57:59 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Attila Nagy <bra@fsn.hu> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: High CPU usage with newnfs(d) - seems to be a cache issue Message-ID: <1721695444.20803113.1378767479011.JavaMail.root@uoguelph.ca> In-Reply-To: <522E4AC5.4040606@fsn.hu>
next in thread | previous in thread | raw e-mail | index | archive | help
Attila Nagy wrote: > Hi, > > I've observed some insane CPU usage on stable/9@r255367. > About the machine: > CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2400.14-MHz > K8-class CPU) > real memory = 34359738368 (32768 MB) > FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads > > It does some NFS serving like this (now running oldnfs) -not quite > peak > times actually: > # nfsstat -w 1 -os > GtAttr Lookup Rdlink Read Write Rename Access Rddir > 763 7206 1 175 92 0 915 3589 > 748 7665 10 131 60 0 905 2923 > 787 9657 23 204 50 0 974 2387 > 517 9881 9 150 41 0 572 2321 > 709 8708 71 235 70 0 1220 3271 > 621 9157 9 254 208 0 928 2563 > 699 5336 29 271 103 0 1242 3448 > 656 4291 11 201 209 0 1119 3908 > 506 3722 0 215 183 0 970 2516 > 698 1476 1 151 66 0 903 2094 > 501 2865 11 268 117 0 995 1392 > 638 6284 46 233 47 0 1096 4847 > 893 7909 47 175 73 0 870 4070 > 651 3936 48 255 51 0 955 2514 > 424 4211 17 223 29 0 745 1458 > 589 8197 26 199 39 0 918 2983 > > It's being hammered by about 40 machines on multiple connections (it > has > 35 UFS file systems exported). > > When running newnfs (admittedly in some stupid way, with -n 32, the > profiling was made with this, maybe this causes some lock > contention), > it occasionally eats 1600% CPU (means: 0 idle). > Lowering the thread number doesn't really solves the problem, I've > seen > -n X*100 CPU usage peaks lately on machines with lower (4-8) -n > counts... > > Doing a profiling with pmc shows that most of the time is spent in > nfsrvd_updatecache and nfsrvd_getcache: > http://pastebin.com/knyppv4d > > Switching back to oldnfsd (even with -n 32) gives a stable 50-60% CPU > usage (out of the "possible" 1600%) when loaded. > > I know that there are some changes regarding this cache in the > CURRENT > code (along with the possibility to set some values with sysctls), > but I > can't run CURRENT. > > Any ideas on how to improve newnfsd, so we can continue serving NFS > in > the future days, where I won't be able to switch back to the old one? > :) > Well, I put a 1 month MFC on r254337 (which I believe fixes this), so it should be in stable/9 in about a week. Alternately, an uglier (but semantically equivalent) patch can be found at: http://people.freebsd.org/~rmacklem/drc4-stable9.patch rick > Thanks, > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1721695444.20803113.1378767479011.JavaMail.root>