Date: Sun, 19 Jan 2014 12:58:25 -0500 From: Adam McDougall <mcdouga9@egr.msu.edu> To: freebsd-net@freebsd.org Subject: Re: Terrible NFS performance under 9.2-RELEASE? Message-ID: <52DC1241.7010004@egr.msu.edu> In-Reply-To: <52DBE1F0.5000507@freebsd.org> References: <CABXB=RSmUe60e%2BJ3bFVOGNcW8B6xyO5Kdgdhbo=3b94tJKUM4w@mail.gmail.com> <52DBE1F0.5000507@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Also try rsize=32768,wsize=32768 in your mount options, made a huge difference for me. I've noticed slow file transfers on NFS in 9 and finally did some searching a couple months ago, someone suggested it and they were on to something. On 01/19/2014 09:32, Alfred Perlstein wrote: > 9.x has pretty poor mbuf tuning by default. > > I hit nearly the same problem and raising the mbufs worked for me. > > I'd suggest raising that and retrying. > > -Alfred > > On 1/19/14 12:47 AM, J David wrote: >> While setting up a test for other purposes, I noticed some really >> horrible NFS performance issues. >> >> To explore this, I set up a test environment with two FreeBSD >> 9.2-RELEASE-p3 virtual machines running under KVM. The NFS server is >> configured to serve a 2 gig mfs on /mnt. >> >> The performance of the virtual network is outstanding: >> >> Server: >> >> $ iperf -c 172.20.20.169 >> >> ------------------------------------------------------------ >> >> Client connecting to 172.20.20.169, TCP port 5001 >> >> TCP window size: 1.00 MByte (default) >> >> ------------------------------------------------------------ >> >> [ 3] local 172.20.20.162 port 59717 connected with 172.20.20.169 port >> 5001 >> >> [ ID] Interval Transfer Bandwidth >> >> [ 3] 0.0-10.0 sec 16.1 GBytes 13.8 Gbits/sec >> >> $ iperf -s >> >> ------------------------------------------------------------ >> >> Server listening on TCP port 5001 >> >> TCP window size: 1.00 MByte (default) >> >> ------------------------------------------------------------ >> >> [ 4] local 172.20.20.162 port 5001 connected with 172.20.20.169 port >> 45655 >> >> [ ID] Interval Transfer Bandwidth >> >> [ 4] 0.0-10.0 sec 15.8 GBytes 13.6 Gbits/sec >> >> >> Client: >> >> >> $ iperf -s >> >> ------------------------------------------------------------ >> >> Server listening on TCP port 5001 >> >> TCP window size: 1.00 MByte (default) >> >> ------------------------------------------------------------ >> >> [ 4] local 172.20.20.169 port 5001 connected with 172.20.20.162 port >> 59717 >> >> [ ID] Interval Transfer Bandwidth >> >> [ 4] 0.0-10.0 sec 16.1 GBytes 13.8 Gbits/sec >> >> ^C$ iperf -c 172.20.20.162 >> >> ------------------------------------------------------------ >> >> Client connecting to 172.20.20.162, TCP port 5001 >> >> TCP window size: 1.00 MByte (default) >> >> ------------------------------------------------------------ >> >> [ 3] local 172.20.20.169 port 45655 connected with 172.20.20.162 port >> 5001 >> >> [ ID] Interval Transfer Bandwidth >> >> [ 3] 0.0-10.0 sec 15.8 GBytes 13.6 Gbits/sec >> >> >> The performance of the mfs filesystem on the server is also good. >> >> Server: >> >> $ sudo mdconfig -a -t swap -s 2g >> >> md0 >> >> $ sudo newfs -U -b 4k -f 4k /dev/md0 >> >> /dev/md0: 2048.0MB (4194304 sectors) block size 4096, fragment size 4096 >> >> using 43 cylinder groups of 48.12MB, 12320 blks, 6160 inodes. >> >> with soft updates >> >> super-block backups (for fsck_ffs -b #) at: >> >> 144, 98704, 197264, 295824, 394384, 492944, 591504, 690064, 788624, >> 887184, >> >> 985744, 1084304, 1182864, 1281424, 1379984, 1478544, 1577104, 1675664, >> >> 1774224, 1872784, 1971344, 2069904, 2168464, 2267024, 2365584, 2464144, >> >> 2562704, 2661264, 2759824, 2858384, 2956944, 3055504, 3154064, 3252624, >> >> 3351184, 3449744, 3548304, 3646864, 3745424, 3843984, 3942544, 4041104, >> >> 4139664 >> >> $ sudo mount /dev/md0 /mnt >> >> $ cd /mnt >> >> $ sudo iozone -e -I -s 512m -r 4k -i 0 -i 1 -i 2 >> >> Iozone: Performance Test of File I/O >> >> Version $Revision: 3.420 $ >> >> [...] >> >> random >> random >> >> KB reclen write rewrite read reread read >> write >> >> 524288 4 560145 1114593 933699 831902 56347 >> 158904 >> >> >> iozone test complete. >> >> >> But introduce NFS into the mix and everything falls apart. >> >> Client: >> >> $ sudo mount -o tcp,nfsv3 f12.phxi:/mnt /mnt >> >> $ cd /mnt >> >> $ sudo iozone -e -I -s 512m -r 4k -i 0 -i 1 -i 2 >> >> Iozone: Performance Test of File I/O >> >> Version $Revision: 3.420 $ >> >> [...] >> >> random >> random >> >> KB reclen write rewrite read reread read >> write >> >> 524288 4 67246 2923 103295 1272407 172475 >> 196 >> >> >> And the above took 48 minutes to run, compared to 14 seconds for the >> local version. So it's 200x slower over NFS. The random write test >> is over 800x slower. Of course NFS is slower, that's expected, but it >> definitely wasn't this exaggerated in previous releases. >> >> To emphasize that iozone reflects real workloads here, I tried doing >> an svn co of the 9-STABLE source tree over NFS but after two hours it >> was still in llvm so I gave up. >> >> While all this not-much-of-anything NFS traffic is going on, both >> systems are essentially idle. The process on the client sits in >> "newnfs" wait state with nearly no CPU. The server is completely idle >> except for the occasional 0.10% in an nfsd thread, which otherwise >> spend their lives in rpcsvc wait state. >> >> Server iostat: >> >> $ iostat -x -w 10 md0 >> >> extended device statistics >> >> device r/s w/s kr/s kw/s qlen svc_t %b >> >> [...] >> >> md0 0.0 36.0 0.0 0.0 0 1.2 0 >> md0 0.0 38.8 0.0 0.0 0 1.5 0 >> md0 0.0 73.6 0.0 0.0 0 1.0 0 >> md0 0.0 53.3 0.0 0.0 0 2.5 0 >> md0 0.0 33.7 0.0 0.0 0 1.1 0 >> md0 0.0 45.5 0.0 0.0 0 1.8 0 >> >> Server nfsstat: >> >> $ nfsstat -s -w 10 >> >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> >> [...] >> >> 0 0 0 471 816 0 0 0 >> >> 0 0 0 480 751 0 0 0 >> >> 0 0 0 481 36 0 0 0 >> >> 0 0 0 469 550 0 0 0 >> >> 0 0 0 485 814 0 0 0 >> >> 0 0 0 467 503 0 0 0 >> >> 0 0 0 473 345 0 0 0 >> >> >> Client nfsstat: >> >> $ nfsstat -c -w 10 >> >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> >> [...] >> >> 0 0 0 0 518 0 0 0 >> >> 0 0 0 0 498 0 0 0 >> >> 0 0 0 0 503 0 0 0 >> >> 0 0 0 0 474 0 0 0 >> >> 0 0 0 0 525 0 0 0 >> >> 0 0 0 0 497 0 0 0 >> >> >> Server vmstat: >> >> $ vmstat -w 10 >> >> procs memory page disks >> faults cpu >> >> r b w avm fre flt re pi po fr sr vt0 vt1 in sy >> cs us sy id >> >> [...] >> >> 0 4 0 634M 6043M 37 0 0 0 1 0 0 0 1561 46 >> 3431 0 2 98 >> >> 0 4 0 640M 6042M 62 0 0 0 28 0 0 0 1598 94 >> 3552 0 2 98 >> >> 0 4 0 648M 6042M 38 0 0 0 0 0 0 0 1609 47 >> 3485 0 1 99 >> >> 0 4 0 648M 6042M 37 0 0 0 0 0 0 0 1615 46 >> 3667 0 2 98 >> >> 0 4 0 648M 6042M 37 0 0 0 0 0 0 0 1606 45 >> 3678 0 2 98 >> >> 0 4 0 648M 6042M 37 0 0 0 0 0 1 0 1561 45 >> 3377 0 2 98 >> >> >> Client vmstat: >> >> $ vmstat -w 10 >> >> procs memory page disks >> faults cpu >> >> r b w avm fre flt re pi po fr sr md0 da0 in sy >> cs us sy id >> >> [...] >> >> 0 0 0 639M 593M 33 0 0 0 1237 0 0 0 281 5575 >> 1043 0 3 97 >> >> 0 0 0 639M 591M 0 0 0 0 712 0 0 0 235 122 >> 889 0 2 98 >> >> 0 0 0 639M 583M 0 0 0 0 571 0 0 1 227 120 >> 851 0 2 98 >> >> 0 0 0 639M 592M 198 0 0 0 1212 0 0 0 251 2497 >> 950 0 3 97 >> >> 0 0 0 639M 586M 0 0 0 0 614 0 0 0 250 121 >> 924 0 2 98 >> >> 0 0 0 639M 586M 0 0 0 0 765 0 0 0 250 120 >> 918 0 3 97 >> >> >> Top on the KVM host says it is 93-95% idle and that each VM sits >> around 7-10% CPU. So basically nobody is doing anything. There's no >> visible bottleneck, and I've no idea where to go from here to figure >> out what's going on. >> >> Does anyone have any suggestions for debugging this? >> >> Thanks! >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52DC1241.7010004>