Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 14 Dec 2011 10:40:03 -0500 (EST)
From:      Rick Macklem <rmacklem@uoguelph.ca>
To:        Peter Maloney <peter.maloney@brockmann-consult.de>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: FreeBSD 9.0 and NFS async -off topic about ZFS ZIL devices
Message-ID:  <1834792825.201945.1323877203185.JavaMail.root@erie.cs.uoguelph.ca>
In-Reply-To: <4EE853F4.7060003@brockmann-consult.de>

next in thread | previous in thread | raw e-mail | index | archive | help
Peter Maloney wrote:
> Am 14.12.2011 01:38, schrieb Rick Macklem:
> > Johan Hendriks wrote:
> >> Rick Macklem schreef:
> >>> Johan Hendriks wrote:
> >>>> Hello all.
> >>>>
> >>>> I used to use async on my 8.x nfs servers!
> >>>> On the FreeBSD 9.0 server i can not do it through the old 8.x
> >>>> sysctl.
> >>>>
> >>>> Is there an other way to set async on FreeBSD 9.x
> >>>>
> >>> You have two choices:
> >>> 1 - Apply this patch to your NFS server's kernel sources and then
> >>> set
> >>>      vfs.nfsd.async=1
> >>>      http://people.freebsd.org/~rmacklem/async.patch
> >>>
> >>> 2 - switch to using the old server by setting
> >>>      oldnfs_server_enable="YES"
> >>>      in your /etc/rc.conf and then setting the sysctl.
> >>>
> >>> I'll assume that you realize that doing this violates the NFS RFCs
> >>> because
> >>> it runs your server in a way where there is a risk of data loss
> >>> (that the
> >>> client won't know to re-write) when the server crashes.
> >>>
> >>> rick
> >>>> regards,
> >>>> Johan Hendriks
> >>>> _______________________________________________
> >>>> freebsd-fs@freebsd.org mailing list
> >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> >>>> To unsubscribe, send any mail to
> >>>> "freebsd-fs-unsubscribe@freebsd.org"
> >> Yes i do know the risk.
> >>
> >> The thing is we want a dataset shared to a ESXi client using NFS.
> >> I use NFS for my normal usage, (sharing ports tree and so on.) but
> >> now
> >> we want to use it to share a ZFS dataset for a ESXi client.
> >> We use iscsi now, but this way we miss some zfs goodies. like
> >> snapshots(not a zvol) and most important, we can reach the files
> >> directly.
> >>
> >> But with a virtual machine shared over NFS i get horrible
> >> performance.
> >> If i copy a file to whatever virtual machine from a windows client
> >> shared with iscsi , i get arround 80Mb per second (in the windows
> >> copy
> >> window) almost at a steady pace. we are really pleased with that.
> >> !!
> >> If i copy a virtual machine to the NFS share, fire it up, and do a
> >> file
> >> copy, it never gets higher than 50 Mb and it sometimes drop to 1 Mb
> >> then
> >> goes to 20 back to 10 and so on.
> >> Also the machines feels sluggish in performance.
> >>
> >> Are there other less dangerous things i can try to boost
> >> performance?
> >>
> > I don't use ZFS, but others have reported using a dedicated SSD that
> > has
> > good write performance for the ZIL log in order to get better write
> > performance for ZFS.
> Indeed, I can get 65 MB/s using a consumer SSD with normal sync nfs
> clients. But once you use ESXi's NFS client, it will drop to somewhere
> between 5 and 9 MB/s. Mine currently goes around 7 MB/s. I also tried
> adding a ramdisk as my ZIL, and then it still only goes 80 MB/s, even
> though obviously the ramdisk should be able to handle random writes
> better than anything, and should go GB/s if there was no other
> bottleneck, not only 80 MB/s (over 10Gbps network, which is tested at
> about 6 Gbps with 1500 MTU). Anything else has to go through RAM at
> some
> point anyway. Unfortunatlely, I didn't test a normal sync nfs client
> with the ramdisk... I'll put that on my mental todo list. I also tried
> a
> UFS zvol, whidh was not using the log, but only went somewhere between
> 40 and 60 MB/s.
> 
It might be interesting to take a look at the Write RPC requests generated
by the ESXi client in wireshark. I wouldn't be surprised if it:
#1 - specifies FILE_SYNC for all writes (this is telling the server it "must"
  commit the data to stable storage before replying and the "hacks" you
  are discussing make the server lie about/ignore this request)
#2 - uses a relatively small block size (if there is a way to tell the
  ESXi client to use a larger block size, I would try that)

If I'm correct w.r.t. the above assumptions, especially #1, all I can
say is that the ESXi client is poorly implemented (or worse;-).

rick

> Here is a very nice thread with some scores with various SSDs using a
> normal NFS client.
> http://forums.freebsd.org/showthread.php?t=23566
> 
> Here is my post in the same thread comparing the linux NFS client with
> sync option, to the ESXi client (writing to a virtual disk filesystem
> with dd):
> http://forums.freebsd.org/showpost.php?p=157154&postcount=55
> 
> 
> >> regards,
> >> Johan Hendriks
> > _______________________________________________
> > freebsd-fs@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to
> > "freebsd-fs-unsubscribe@freebsd.org"
> 
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1834792825.201945.1323877203185.JavaMail.root>