Date: Wed, 2 Jul 2014 20:04:06 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: "Russell L. Carter" <rcarter@pinyon.org> Cc: freebsd-net@freebsd.org Subject: Re: NFS client READ performance on -current Message-ID: <371130768.6608219.1404345846086.JavaMail.root@uoguelph.ca> In-Reply-To: <53B49AE0.4030902@pinyon.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Russell L. Carter wrote: > Greetings! > It's been 14 years. OMG do I love poudriere and zfs. > > But apropos of this post from last January: > > http://lists.freebsd.org/pipermail/freebsd-net/2014-January/037547.html > > I am going to capitalize READ to emphasize that all I am looking at > here is reading on the client. In some of posts I have read people > have been confusing read vs. write performance and losing the thread. > > I have fresh current server and client, each with otherwise quiescent > em NICs. I also have debian kernel 3.14-1 server and client. The > freebsd hardware is large enough, 16GB RAM + vishera 8 cores. (The > debian systems are much smaller/slower) > > Upshot is that with a -current NFSV4 (4.1) serving a -current client > and a linux client, I see today client READ performance of: > > -current -> -current: 2.7MB/s READ nice and steady > -current -> linux: 80MB/s READ very bursty > > And for the weird (to me) result: > > linux -> -current: 55.74MB/s READ > > So to be very clear, the freebsd -current server seems capable of of > supporting excellent client READ bandwidth. But something is > dramatically awry with the freebsd client. (scp, rsync over ssh, all > are near wire speed when I get rid of disk overhead). The -current > server has got an 11TB raidz2 behind it but I'm only READing, not > writing, so all of the discussion about ZFS + NFS sync write problems > seems immaterial, possibly. > > Here's the mount info: > > -current client's output from nfsstat -m (terp is current server, ari > is linux server): > > terp:/raid/library on /mnt/terp/library > nfsv4,minorversion=1,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=2798255,timeout=120,retrans=2147483647 > > ari:/d1/library on /mnt/ari > nfsv4,minorversion=0,tcp,resvport,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=2798255,timeout=120,retrans=2147483647 > > Linux client's excerpt from mount output: > > terp:/raid/library on /mnt/terp/library type nfs4 > (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.10.3,local_lock=none,addr=10.0.10.4) > > BTW, linux -> linux is about 80MB/s > > I tried nfsv3 -current -> -current and still got 2.7MB/s READ. > > I have spent about 5 hours strolling through the archives of the last > several years of discussion about this issue in various forums, but I > didn't see a reference once since Jaunary's thread on freebsd-net, > and > I didn't find a bug entry, so I thought I'd keep it alive, so to > speak. > > I've tried all of the various sysctl tweaks that have been suggested > over time, that would make sense for reads, to no effect. > Please try disabling TSO on the FreeBSD systems and also try an rsize=32768 to see if either of those have any effect. There was also a recent case where disabling msix interrupt handling for the network interface fixed a slow NFS I/O rate issue. (I don't know what the loader conf variable was, but look for something with msix in it for the net device driver. It might be hw.em.enable_msix=0 or something like that?) rick > Russell > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?371130768.6608219.1404345846086.JavaMail.root>