Date: Thu, 29 Jan 2009 00:43:05 -0800 From: Brent Jones <brent@servuhome.net> To: freebsd-performance@freebsd.org, pathiaki2@yahoo.com, freebsd-stable@freebsd.org Subject: Re: ZFS, NFS and Network tuning Message-ID: <ee9f3b480901290043v54a547bk678458ed36887ec2@mail.gmail.com> In-Reply-To: <ee9f3b480901282321h5b57fa49ud39265c7523e0cdf@mail.gmail.com> References: <ee9f3b480901282321h5b57fa49ud39265c7523e0cdf@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jan 28, 2009 at 11:21 PM, Brent Jones <brent@servuhome.net> wrote: > I'm reviving this, as I too am seeing something eerily similar. I have > made my own thread under freebsd-stable, so I will hopefully move that > discussion to this list. > > I believe we are seeing performance problems when the FreeBSD NFS > client issues FSYNC NFS instead of ASYNC, sending performance to a > mere percentage of what disks and network links are capable of. > Further testing tonight demonstrates that other NFSv3 and v4 clients > do not issue FSYNC unless they modify attributed and close a file, or > append and close a file. > FreeBSD NFS client will issue FSYNCs anytime the write size (-w) is > reached, instead of when just closing the file. > This is not necessary, since NFSv3 and v4 TCP have provisions for safe > async writes that 'guarantee' state of NFS writes. > > Here is the contents of what I wrote there verbatim: > > http://lists.freebsd.org/pipermail/freebsd-stable/2009-January/048063.html > > ------- > > > Hello FreeBSD users, > I am running into some performance problems with NFSv3/v4 mounts. > I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares > The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram > and a FreeBSD 7.1 32bit with same hardware. > > The issue I am seeing, is that for certain file types, the FreeBSD NFS > client will either issue an ASYNC write, or an FSYNC. > However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP > versions of the protocol, so that should be the default. > Issuing FSYNC's for every compete block transmitted adds substantial > overhead and slows everything down. > > The two test files I have that can reproduce this data are a file > created by 'dump' which is just binary data: > > $ file testbinery > testbinery: data > > ASCII text file from a Maildir format: > > $ file ascittest > ascittest: ASCII mail text > > My NFS mount command lines I have tried to get all data to ASYNC write: > > $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > > Here is an excerpt from a snoop from the binary data file: > > $ snoop rpc nfs > > obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 > (read,lookup,modify,extend,delete,execute) > pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 testbinery > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3 > obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 > (read,lookup,modify,extend,delete,execute) > pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) > obsmtp02.local -> pdxfilu01 NFS C SETATTR3 FH=57D3 > pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 582647808 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 592871424 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 605421568 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > > > And on and on.. it will acheive near full wire-speed, about 110MB/sec > during the copy > > > Here is the same snoop, only copying the ASCII mail file: > > $ snoop rpc nfs > > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory > obsmtp02.local -> pdxfilu01 NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest > pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3 > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > > > And so on. I've reproduced this with several files, and the only > difference between tests is the file type. > Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the > file type/contents? > If so, is there a tuneable setting to make all write ASYNC? > Otherwise, FSYNC'ing for every block written over NFS will cause so > many IOPS on the NFS server, that performance will degrade severely. > > Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for > any file type, if mounted with NFSv3 of NFSv4 (TCP). > > Any ideas? > > Thanks in advance! > > > > > -- > Brent Jones > brent@servuhome.net > I have found a 4 year old bug, which may be related to this. cp uses mmap for small files (and I imagine lots of things use mmap for file operations) and causes slowdowns via NFS, due to the fsync data provided above. http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/87792 That bugid accurately describes the issue, is there any way to attach more 'interested parties' or additional details to that bug? -- Brent Jones brent@servuhome.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ee9f3b480901290043v54a547bk678458ed36887ec2>