From owner-freebsd-fs@FreeBSD.ORG Wed Dec 14 15:40:09 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1335010656D1 for ; Wed, 14 Dec 2011 15:40:09 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 42F7F8FC13 for ; Wed, 14 Dec 2011 15:40:03 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAHDC6E6DaFvO/2dsb2JhbAApGhaEcqckgXIBAQEDAQEBASArIAsFFhgCAg0ZAikBCSYGCAcEARwEh1kII6UWkgSBL4JKhnqBFgSIMoodgiWSSA X-IronPort-AV: E=Sophos;i="4.71,352,1320642000"; d="scan'208";a="150268748" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 14 Dec 2011 10:40:03 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 311CFB3F2C; Wed, 14 Dec 2011 10:40:03 -0500 (EST) Date: Wed, 14 Dec 2011 10:40:03 -0500 (EST) From: Rick Macklem To: Peter Maloney Message-ID: <1834792825.201945.1323877203185.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <4EE853F4.7060003@brockmann-consult.de> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 9.0 and NFS async -off topic about ZFS ZIL devices X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 15:40:09 -0000 Peter Maloney wrote: > Am 14.12.2011 01:38, schrieb Rick Macklem: > > Johan Hendriks wrote: > >> Rick Macklem schreef: > >>> Johan Hendriks wrote: > >>>> Hello all. > >>>> > >>>> I used to use async on my 8.x nfs servers! > >>>> On the FreeBSD 9.0 server i can not do it through the old 8.x > >>>> sysctl. > >>>> > >>>> Is there an other way to set async on FreeBSD 9.x > >>>> > >>> You have two choices: > >>> 1 - Apply this patch to your NFS server's kernel sources and then > >>> set > >>> vfs.nfsd.async=1 > >>> http://people.freebsd.org/~rmacklem/async.patch > >>> > >>> 2 - switch to using the old server by setting > >>> oldnfs_server_enable="YES" > >>> in your /etc/rc.conf and then setting the sysctl. > >>> > >>> I'll assume that you realize that doing this violates the NFS RFCs > >>> because > >>> it runs your server in a way where there is a risk of data loss > >>> (that the > >>> client won't know to re-write) when the server crashes. > >>> > >>> rick > >>>> regards, > >>>> Johan Hendriks > >>>> _______________________________________________ > >>>> freebsd-fs@freebsd.org mailing list > >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>>> To unsubscribe, send any mail to > >>>> "freebsd-fs-unsubscribe@freebsd.org" > >> Yes i do know the risk. > >> > >> The thing is we want a dataset shared to a ESXi client using NFS. > >> I use NFS for my normal usage, (sharing ports tree and so on.) but > >> now > >> we want to use it to share a ZFS dataset for a ESXi client. > >> We use iscsi now, but this way we miss some zfs goodies. like > >> snapshots(not a zvol) and most important, we can reach the files > >> directly. > >> > >> But with a virtual machine shared over NFS i get horrible > >> performance. > >> If i copy a file to whatever virtual machine from a windows client > >> shared with iscsi , i get arround 80Mb per second (in the windows > >> copy > >> window) almost at a steady pace. we are really pleased with that. > >> !! > >> If i copy a virtual machine to the NFS share, fire it up, and do a > >> file > >> copy, it never gets higher than 50 Mb and it sometimes drop to 1 Mb > >> then > >> goes to 20 back to 10 and so on. > >> Also the machines feels sluggish in performance. > >> > >> Are there other less dangerous things i can try to boost > >> performance? > >> > > I don't use ZFS, but others have reported using a dedicated SSD that > > has > > good write performance for the ZIL log in order to get better write > > performance for ZFS. > Indeed, I can get 65 MB/s using a consumer SSD with normal sync nfs > clients. But once you use ESXi's NFS client, it will drop to somewhere > between 5 and 9 MB/s. Mine currently goes around 7 MB/s. I also tried > adding a ramdisk as my ZIL, and then it still only goes 80 MB/s, even > though obviously the ramdisk should be able to handle random writes > better than anything, and should go GB/s if there was no other > bottleneck, not only 80 MB/s (over 10Gbps network, which is tested at > about 6 Gbps with 1500 MTU). Anything else has to go through RAM at > some > point anyway. Unfortunatlely, I didn't test a normal sync nfs client > with the ramdisk... I'll put that on my mental todo list. I also tried > a > UFS zvol, whidh was not using the log, but only went somewhere between > 40 and 60 MB/s. > It might be interesting to take a look at the Write RPC requests generated by the ESXi client in wireshark. I wouldn't be surprised if it: #1 - specifies FILE_SYNC for all writes (this is telling the server it "must" commit the data to stable storage before replying and the "hacks" you are discussing make the server lie about/ignore this request) #2 - uses a relatively small block size (if there is a way to tell the ESXi client to use a larger block size, I would try that) If I'm correct w.r.t. the above assumptions, especially #1, all I can say is that the ESXi client is poorly implemented (or worse;-). rick > Here is a very nice thread with some scores with various SSDs using a > normal NFS client. > http://forums.freebsd.org/showthread.php?t=23566 > > Here is my post in the same thread comparing the linux NFS client with > sync option, to the ESXi client (writing to a virtual disk filesystem > with dd): > http://forums.freebsd.org/showpost.php?p=157154&postcount=55 > > > >> regards, > >> Johan Hendriks > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to > > "freebsd-fs-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"