From owner-freebsd-stable@FreeBSD.ORG Thu Jan 29 03:38:05 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07328106564A for ; Thu, 29 Jan 2009 03:38:05 +0000 (UTC) (envelope-from brent@servuhome.net) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.157]) by mx1.freebsd.org (Postfix) with ESMTP id 7F8468FC16 for ; Thu, 29 Jan 2009 03:38:03 +0000 (UTC) (envelope-from brent@servuhome.net) Received: by fg-out-1718.google.com with SMTP id e12so961021fga.35 for ; Wed, 28 Jan 2009 19:38:03 -0800 (PST) MIME-Version: 1.0 Received: by 10.181.17.13 with SMTP id u13mr1073746bki.152.1233198591991; Wed, 28 Jan 2009 19:09:51 -0800 (PST) Date: Wed, 28 Jan 2009 19:09:51 -0800 Message-ID: From: Brent Jones To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: NFS writes calling FSYNC and ASYNC not consistent X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jan 2009 03:38:05 -0000 Hello FreeBSD users, I am running into some performance problems with NFSv3/v4 mounts. I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram and a FreeBSD 7.1 32bit with same hardware. The issue I am seeing, is that for certain file types, the FreeBSD NFS client will either issue an ASYNC write, or an FSYNC. However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP versions of the protocol, so that should be the default. Issuing FSYNC's for every compete block transmitted adds substantial overhead and slows everything down. The two test files I have that can reproduce this data are a file created by 'dump' which is just binary data: $ file testbinery testbinery: data ASCII text file from a Maildir format: $ file ascittest ascittest: ASCII mail text My NFS mount command lines I have tried to get all data to ASYNC write: $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ Here is an excerpt from a snoop from the binary data file: $ snoop rpc nfs obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 (read,lookup,modify,extend,delete,execute) pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 testbinery pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3 obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 (read,lookup,modify,extend,delete,execute) pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) obsmtp02.local -> pdxfilu01 NFS C SETATTR3 FH=57D3 pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 582647808 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 592871424 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 605421568 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) And on and on.. it will acheive near full wire-speed, about 110MB/sec during the copy Here is the same snoop, only copying the ASCII mail file: $ snoop rpc nfs obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory obsmtp02.local -> pdxfilu01 NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3 obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) And so on. I've reproduced this with several files, and the only difference between tests is the file type. Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the file type/contents? If so, is there a tuneable setting to make all write ASYNC? Otherwise, FSYNC'ing for every block written over NFS will cause so many IOPS on the NFS server, that performance will degrade severely. Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for any file type, if mounted with NFSv3 of NFSv4 (TCP). Any ideas? Thanks in advance! -- Brent Jones brent@servuhome.net