From owner-freebsd-performance@FreeBSD.ORG Thu Jan 29 08:43:08 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2AD971065673; Thu, 29 Jan 2009 08:43:08 +0000 (UTC) (envelope-from brent@servuhome.net) Received: from mail-fx0-f15.google.com (mail-fx0-f15.google.com [209.85.220.15]) by mx1.freebsd.org (Postfix) with ESMTP id 6AEF58FC12; Thu, 29 Jan 2009 08:43:07 +0000 (UTC) (envelope-from brent@servuhome.net) Received: by fxm8 with SMTP id 8so761424fxm.19 for ; Thu, 29 Jan 2009 00:43:06 -0800 (PST) MIME-Version: 1.0 Received: by 10.181.199.16 with SMTP id b16mr583902bkq.142.1233218586010; Thu, 29 Jan 2009 00:43:06 -0800 (PST) In-Reply-To: References: Date: Thu, 29 Jan 2009 00:43:05 -0800 Message-ID: From: Brent Jones To: freebsd-performance@freebsd.org, pathiaki2@yahoo.com, freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: Re: ZFS, NFS and Network tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2009 08:43:08 -0000 On Wed, Jan 28, 2009 at 11:21 PM, Brent Jones wrote: > I'm reviving this, as I too am seeing something eerily similar. I have > made my own thread under freebsd-stable, so I will hopefully move that > discussion to this list. > > I believe we are seeing performance problems when the FreeBSD NFS > client issues FSYNC NFS instead of ASYNC, sending performance to a > mere percentage of what disks and network links are capable of. > Further testing tonight demonstrates that other NFSv3 and v4 clients > do not issue FSYNC unless they modify attributed and close a file, or > append and close a file. > FreeBSD NFS client will issue FSYNCs anytime the write size (-w) is > reached, instead of when just closing the file. > This is not necessary, since NFSv3 and v4 TCP have provisions for safe > async writes that 'guarantee' state of NFS writes. > > Here is the contents of what I wrote there verbatim: > > http://lists.freebsd.org/pipermail/freebsd-stable/2009-January/048063.html > > ------- > > > Hello FreeBSD users, > I am running into some performance problems with NFSv3/v4 mounts. > I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares > The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram > and a FreeBSD 7.1 32bit with same hardware. > > The issue I am seeing, is that for certain file types, the FreeBSD NFS > client will either issue an ASYNC write, or an FSYNC. > However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP > versions of the protocol, so that should be the default. > Issuing FSYNC's for every compete block transmitted adds substantial > overhead and slows everything down. > > The two test files I have that can reproduce this data are a file > created by 'dump' which is just binary data: > > $ file testbinery > testbinery: data > > ASCII text file from a Maildir format: > > $ file ascittest > ascittest: ASCII mail text > > My NFS mount command lines I have tried to get all data to ASYNC write: > > $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ > > Here is an excerpt from a snoop from the binary data file: > > $ snoop rpc nfs > > obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 > (read,lookup,modify,extend,delete,execute) > pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 testbinery > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3 > obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 > (read,lookup,modify,extend,delete,execute) > pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) > obsmtp02.local -> pdxfilu01 NFS C SETATTR3 FH=57D3 > pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 582647808 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 592871424 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 605421568 for > 32768 (ASYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) > > > And on and on.. it will acheive near full wire-speed, about 110MB/sec > during the copy > > > Here is the same snoop, only copying the ASCII mail file: > > $ snoop rpc nfs > > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory > obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest > pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory > obsmtp02.local -> pdxfilu01 NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest > pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3 > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC) > pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) > > > And so on. I've reproduced this with several files, and the only > difference between tests is the file type. > Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the > file type/contents? > If so, is there a tuneable setting to make all write ASYNC? > Otherwise, FSYNC'ing for every block written over NFS will cause so > many IOPS on the NFS server, that performance will degrade severely. > > Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for > any file type, if mounted with NFSv3 of NFSv4 (TCP). > > Any ideas? > > Thanks in advance! > > > > > -- > Brent Jones > brent@servuhome.net > I have found a 4 year old bug, which may be related to this. cp uses mmap for small files (and I imagine lots of things use mmap for file operations) and causes slowdowns via NFS, due to the fsync data provided above. http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/87792 That bugid accurately describes the issue, is there any way to attach more 'interested parties' or additional details to that bug? -- Brent Jones brent@servuhome.net