Date: Sun, 19 Jan 2014 06:32:16 -0800 From: Alfred Perlstein <alfred@freebsd.org> To: freebsd-net@freebsd.org Subject: Re: Terrible NFS performance under 9.2-RELEASE? Message-ID: <52DBE1F0.5000507@freebsd.org> In-Reply-To: <CABXB=RSmUe60e%2BJ3bFVOGNcW8B6xyO5Kdgdhbo=3b94tJKUM4w@mail.gmail.com> References: <CABXB=RSmUe60e%2BJ3bFVOGNcW8B6xyO5Kdgdhbo=3b94tJKUM4w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
9.x has pretty poor mbuf tuning by default. I hit nearly the same problem and raising the mbufs worked for me. I'd suggest raising that and retrying. -Alfred On 1/19/14 12:47 AM, J David wrote: > While setting up a test for other purposes, I noticed some really > horrible NFS performance issues. > > To explore this, I set up a test environment with two FreeBSD > 9.2-RELEASE-p3 virtual machines running under KVM. The NFS server is > configured to serve a 2 gig mfs on /mnt. > > The performance of the virtual network is outstanding: > > Server: > > $ iperf -c 172.20.20.169 > > ------------------------------------------------------------ > > Client connecting to 172.20.20.169, TCP port 5001 > > TCP window size: 1.00 MByte (default) > > ------------------------------------------------------------ > > [ 3] local 172.20.20.162 port 59717 connected with 172.20.20.169 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 16.1 GBytes 13.8 Gbits/sec > > $ iperf -s > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 1.00 MByte (default) > > ------------------------------------------------------------ > > [ 4] local 172.20.20.162 port 5001 connected with 172.20.20.169 port 45655 > > [ ID] Interval Transfer Bandwidth > > [ 4] 0.0-10.0 sec 15.8 GBytes 13.6 Gbits/sec > > > Client: > > > $ iperf -s > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 1.00 MByte (default) > > ------------------------------------------------------------ > > [ 4] local 172.20.20.169 port 5001 connected with 172.20.20.162 port 59717 > > [ ID] Interval Transfer Bandwidth > > [ 4] 0.0-10.0 sec 16.1 GBytes 13.8 Gbits/sec > > ^C$ iperf -c 172.20.20.162 > > ------------------------------------------------------------ > > Client connecting to 172.20.20.162, TCP port 5001 > > TCP window size: 1.00 MByte (default) > > ------------------------------------------------------------ > > [ 3] local 172.20.20.169 port 45655 connected with 172.20.20.162 port 5001 > > [ ID] Interval Transfer Bandwidth > > [ 3] 0.0-10.0 sec 15.8 GBytes 13.6 Gbits/sec > > > The performance of the mfs filesystem on the server is also good. > > Server: > > $ sudo mdconfig -a -t swap -s 2g > > md0 > > $ sudo newfs -U -b 4k -f 4k /dev/md0 > > /dev/md0: 2048.0MB (4194304 sectors) block size 4096, fragment size 4096 > > using 43 cylinder groups of 48.12MB, 12320 blks, 6160 inodes. > > with soft updates > > super-block backups (for fsck_ffs -b #) at: > > 144, 98704, 197264, 295824, 394384, 492944, 591504, 690064, 788624, 887184, > > 985744, 1084304, 1182864, 1281424, 1379984, 1478544, 1577104, 1675664, > > 1774224, 1872784, 1971344, 2069904, 2168464, 2267024, 2365584, 2464144, > > 2562704, 2661264, 2759824, 2858384, 2956944, 3055504, 3154064, 3252624, > > 3351184, 3449744, 3548304, 3646864, 3745424, 3843984, 3942544, 4041104, > > 4139664 > > $ sudo mount /dev/md0 /mnt > > $ cd /mnt > > $ sudo iozone -e -I -s 512m -r 4k -i 0 -i 1 -i 2 > > Iozone: Performance Test of File I/O > > Version $Revision: 3.420 $ > > [...] > > random random > > KB reclen write rewrite read reread read write > > 524288 4 560145 1114593 933699 831902 56347 > 158904 > > > iozone test complete. > > > But introduce NFS into the mix and everything falls apart. > > Client: > > $ sudo mount -o tcp,nfsv3 f12.phxi:/mnt /mnt > > $ cd /mnt > > $ sudo iozone -e -I -s 512m -r 4k -i 0 -i 1 -i 2 > > Iozone: Performance Test of File I/O > > Version $Revision: 3.420 $ > > [...] > > random random > > KB reclen write rewrite read reread read write > > 524288 4 67246 2923 103295 1272407 172475 > 196 > > > And the above took 48 minutes to run, compared to 14 seconds for the > local version. So it's 200x slower over NFS. The random write test > is over 800x slower. Of course NFS is slower, that's expected, but it > definitely wasn't this exaggerated in previous releases. > > To emphasize that iozone reflects real workloads here, I tried doing > an svn co of the 9-STABLE source tree over NFS but after two hours it > was still in llvm so I gave up. > > While all this not-much-of-anything NFS traffic is going on, both > systems are essentially idle. The process on the client sits in > "newnfs" wait state with nearly no CPU. The server is completely idle > except for the occasional 0.10% in an nfsd thread, which otherwise > spend their lives in rpcsvc wait state. > > Server iostat: > > $ iostat -x -w 10 md0 > > extended device statistics > > device r/s w/s kr/s kw/s qlen svc_t %b > > [...] > > md0 0.0 36.0 0.0 0.0 0 1.2 0 > md0 0.0 38.8 0.0 0.0 0 1.5 0 > md0 0.0 73.6 0.0 0.0 0 1.0 0 > md0 0.0 53.3 0.0 0.0 0 2.5 0 > md0 0.0 33.7 0.0 0.0 0 1.1 0 > md0 0.0 45.5 0.0 0.0 0 1.8 0 > > Server nfsstat: > > $ nfsstat -s -w 10 > > GtAttr Lookup Rdlink Read Write Rename Access Rddir > > [...] > > 0 0 0 471 816 0 0 0 > > 0 0 0 480 751 0 0 0 > > 0 0 0 481 36 0 0 0 > > 0 0 0 469 550 0 0 0 > > 0 0 0 485 814 0 0 0 > > 0 0 0 467 503 0 0 0 > > 0 0 0 473 345 0 0 0 > > > Client nfsstat: > > $ nfsstat -c -w 10 > > GtAttr Lookup Rdlink Read Write Rename Access Rddir > > [...] > > 0 0 0 0 518 0 0 0 > > 0 0 0 0 498 0 0 0 > > 0 0 0 0 503 0 0 0 > > 0 0 0 0 474 0 0 0 > > 0 0 0 0 525 0 0 0 > > 0 0 0 0 497 0 0 0 > > > Server vmstat: > > $ vmstat -w 10 > > procs memory page disks faults cpu > > r b w avm fre flt re pi po fr sr vt0 vt1 in sy > cs us sy id > > [...] > > 0 4 0 634M 6043M 37 0 0 0 1 0 0 0 1561 46 > 3431 0 2 98 > > 0 4 0 640M 6042M 62 0 0 0 28 0 0 0 1598 94 > 3552 0 2 98 > > 0 4 0 648M 6042M 38 0 0 0 0 0 0 0 1609 47 > 3485 0 1 99 > > 0 4 0 648M 6042M 37 0 0 0 0 0 0 0 1615 46 > 3667 0 2 98 > > 0 4 0 648M 6042M 37 0 0 0 0 0 0 0 1606 45 > 3678 0 2 98 > > 0 4 0 648M 6042M 37 0 0 0 0 0 1 0 1561 45 > 3377 0 2 98 > > > Client vmstat: > > $ vmstat -w 10 > > procs memory page disks faults cpu > > r b w avm fre flt re pi po fr sr md0 da0 in sy > cs us sy id > > [...] > > 0 0 0 639M 593M 33 0 0 0 1237 0 0 0 281 5575 > 1043 0 3 97 > > 0 0 0 639M 591M 0 0 0 0 712 0 0 0 235 122 > 889 0 2 98 > > 0 0 0 639M 583M 0 0 0 0 571 0 0 1 227 120 > 851 0 2 98 > > 0 0 0 639M 592M 198 0 0 0 1212 0 0 0 251 2497 > 950 0 3 97 > > 0 0 0 639M 586M 0 0 0 0 614 0 0 0 250 121 > 924 0 2 98 > > 0 0 0 639M 586M 0 0 0 0 765 0 0 0 250 120 > 918 0 3 97 > > > Top on the KVM host says it is 93-95% idle and that each VM sits > around 7-10% CPU. So basically nobody is doing anything. There's no > visible bottleneck, and I've no idea where to go from here to figure > out what's going on. > > Does anyone have any suggestions for debugging this? > > Thanks! > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52DBE1F0.5000507>