Date: Tue, 5 Nov 2002 10:48:00 -0500 From: Steve Shorter <steve@nomad.lets.net> To: questions@freebsd.org Cc: d.anker@au.darkbluesea.com Subject: Re: NFS Performance woes Message-ID: <20021105104800.A89144@nomad.lets.net>
next in thread | raw e-mail | index | archive | help
Howdy! I have done some simulations with NFS servers - Intel SCB2 (4G RAM) serving files from 500G RAID devices. I created a treed directory structure with 300G of 32k files that approximates our "homedirectory" structure. I had about 6 diskless front ends (tyan 2518 with 2G) that NFS booted and mounted the "homedir" and ran multiple scripts that walked through the directory structure reading files and writing them to /dev/null. All machines have 3 intel 100 NICs. One interface is used to mount the root etc. and the other is used to mount "homedirs". The NFS server serves root from fxp0 and "homedir" data from both fxp1 fxp2. The diskless front ends mount root from fxp0 and mount "homedir" from fxp1. When the simulation was running full out I was serving about 1/3-1/2 data from page cache and 2/3-1/2 from disk. I tried numerous configurations and tuning of network parameters after doing research and discovered... for FreeBSD 4.6.2 1) NFS over UDP outperforms NFS over TCP. I was able to average about 70Mbs over *both* (occasionally they would almost max out ie. 95Mbs) interfaces serving data using UDP mounts with 8K rw. (the default). No matter what I tried with TCP I never got more than half that throughput. 2) The optimal number of nfsd's to run on the server was about 100! If I reduced the number of nfsds below 80 it would start to choke off the data moving through the network. I found that at around 100 there was no more increase. You must make a minor change to source as the max allowed now by default is 20. I was running 8 nfsiod's on the clients. TCP mounts under tested conditions always had much higher loads than UDP. Also it was impossible to do an "ls" on a mounted directory under load with TCP. With UDP there were no such problems. If you are using UDP it is *essential* that you monitor fragments that are being droppend because of timeout. If you have a good network this should not be a problem. For example I have a webserver that has been up 20 days and has moved 1G of fragments but has only dropped about 800. Also TCP mounts will require a remount of the clients if the server should crash/whatever. UDP just keeps on ticking. If you have Gig ether than there is other tuning you *must* do to realize this potential. Personally I think it is better to use multiple 100Mbs NIC's than to use Gig ether if you can get away with it. YMMV. -steve To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021105104800.A89144>