Date: Tue, 19 Nov 2013 10:12:46 -0800 From: aurfalien <aurfalien@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: Performance difference between UFS and ZFS with NFS Message-ID: <9F76D61C-EFEB-44B3-9717-D0795789832D@gmail.com> In-Reply-To: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> References: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote: > Eric Browning wrote: >> Some background: >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ 3Ghz, >> -64GB ram each server >> -Four Intel DC S3700 800GB SSDs for primary storage, each server. >> -FreeBSD 9 stable as of 902503 >> -ZFS v28 and later updated to feature flags (v29?) >> -LSI 9200-8i controller >> -Intel I350T4 nic (only one port being used currently) using all four >> in >> LACP overtaxed the server's NFS queue from what we found out making >> the >> server basically unusable. >>=20 >> There is definitely something going on between NFS and ZFS when used >> as a >> file server (random workload) for mac home directories. They do not >> jive >> well at all and pretty much drag down these beefy servers and cause >> 20-30 >> second delays when just attempting to list a directory on Mac 10.7, >> 10.8 >> clients although throughput seems fast when copying files. >>=20 >> This server's NFS was sitting north of 700% (7+ cores) all day long >> when >> using ZFSv28 raidz1. I have also tried stripe, compression on/off, >> sync >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. >> I've >> tried just 100% stock settings in loader.conf and and some >> recommended >> tuning from various sources on the freebsd lists and other sites >> including >> the freebsd handbook. >>=20 >> This is my mountpoint creation: >> zfs create -o mountpoint=3D/users -o sharenfs=3Don -o >> casesensitivity=3Dinsensitive -o aclmode=3Dpassthrough -o = compression=3Dlz4 >> -o >> atime=3Doff -o aclinherit=3Dpassthrough tank/users >>=20 >> This last weekend I switched one of these servers over to a UFS raid >> 0 >> setup and NFS now only eats about 36% of one core during the initial >> login >> phase of 150-ish users over about 10 minutes and sits under 1-3% >> during >> normal usage and directories all list instantly even when drilling >> down 10 >> or so directories on the client's home files. The same NFS config on >> server >> and clients are still active. >>=20 >> Right now I'm going to have to abandon ZFS until it works with NFS. >> I >> don't want to get into a finger pointing game, I'd just like to help >> get >> this fixed, I have one old i386 server I can try things out on if >> that >> helps and it's already on 9 stable and ZFS v28. >>=20 > Btw, in previous discussions with Eric on this, he provided nfsstat > output that seemed to indicate most of his RPC load from the Macs > were Access and Getattr RPCs. >=20 > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a > significant part of this issue. I know nothing about ZFS, but I = believe > it does always have ACLs enabled and presumably needs to check the > ACL for each VOP_ACCESSX(). >=20 > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and > VOP_GETATTR() can look at these? Indeed. However couldn't one simply disable ACL mode via; zfs set aclinherit=3Ddiscard pool/dataset zfs set aclmode=3Ddiscard pool/dataset Eric, mind setting these and see? Mid/late this week I'll be doing a rather large render farm test amongst = our Mac fleet against ZFS. Will reply to this thread with outcome when I'm done. Should be = interesting. - aurf =20 >=20 > rick >=20 >> Thanks, >> -- >> Eric Browning >> Systems Administrator >> 801-984-7623 >>=20 >> Skaggs Catholic Center >> Juan Diego Catholic High School >> Saint John the Baptist Middle >> Saint John the Baptist Elementary >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9F76D61C-EFEB-44B3-9717-D0795789832D>