Date: Tue, 19 Nov 2013 10:25:03 -0800 From: aurfalien <aurfalien@gmail.com> To: Eric Browning <ericbrowning@skaggscatholiccenter.org> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: Performance difference between UFS and ZFS with NFS Message-ID: <5969250F-0987-4304-BB95-52C7BAE8D84D@gmail.com> In-Reply-To: <CAM=5oeAF2gfccrGNdbApUDpqRae4OQjZ7oaZZi4y1j%2BsF6PsTw@mail.gmail.com> References: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> <9F76D61C-EFEB-44B3-9717-D0795789832D@gmail.com> <CAM=5oeAF2gfccrGNdbApUDpqRae4OQjZ7oaZZi4y1j%2BsF6PsTw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Curious. Do you have NFS locking enabled client side? Most likely you do as Mac Mail will not run w/o locks, nor will Adobe = prefs like temp cache. etc... So being this is prolly the case, could it be a mem pressure issue and = not enough RAM? So NFS locks take up RAM as does ARC. What are your mem stats and swap = stats during the 700% (yikes) experience? - aurf On Nov 19, 2013, at 10:19 AM, Eric Browning wrote: > Aurf, >=20 > I ran those two commands and it doesn't seem to have made a = difference. Usage is still above 700% and it still takes 30s to list a = directory. The time to list is proportional to the number of users = logged in. On UFS with all students logged in and hammering away at = their files there is no noticeable speed decrease. >=20 >=20 > On Tue, Nov 19, 2013 at 11:12 AM, aurfalien <aurfalien@gmail.com> = wrote: >=20 > On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote: >=20 > > Eric Browning wrote: > >> Some background: > >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ = 3Ghz, > >> -64GB ram each server > >> -Four Intel DC S3700 800GB SSDs for primary storage, each server. > >> -FreeBSD 9 stable as of 902503 > >> -ZFS v28 and later updated to feature flags (v29?) > >> -LSI 9200-8i controller > >> -Intel I350T4 nic (only one port being used currently) using all = four > >> in > >> LACP overtaxed the server's NFS queue from what we found out making > >> the > >> server basically unusable. > >> > >> There is definitely something going on between NFS and ZFS when = used > >> as a > >> file server (random workload) for mac home directories. They do = not > >> jive > >> well at all and pretty much drag down these beefy servers and cause > >> 20-30 > >> second delays when just attempting to list a directory on Mac 10.7, > >> 10.8 > >> clients although throughput seems fast when copying files. > >> > >> This server's NFS was sitting north of 700% (7+ cores) all day long > >> when > >> using ZFSv28 raidz1. I have also tried stripe, compression on/off, > >> sync > >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. > >> I've > >> tried just 100% stock settings in loader.conf and and some > >> recommended > >> tuning from various sources on the freebsd lists and other sites > >> including > >> the freebsd handbook. > >> > >> This is my mountpoint creation: > >> zfs create -o mountpoint=3D/users -o sharenfs=3Don -o > >> casesensitivity=3Dinsensitive -o aclmode=3Dpassthrough -o = compression=3Dlz4 > >> -o > >> atime=3Doff -o aclinherit=3Dpassthrough tank/users > >> > >> This last weekend I switched one of these servers over to a UFS = raid > >> 0 > >> setup and NFS now only eats about 36% of one core during the = initial > >> login > >> phase of 150-ish users over about 10 minutes and sits under 1-3% > >> during > >> normal usage and directories all list instantly even when drilling > >> down 10 > >> or so directories on the client's home files. The same NFS config = on > >> server > >> and clients are still active. > >> > >> Right now I'm going to have to abandon ZFS until it works with NFS. > >> I > >> don't want to get into a finger pointing game, I'd just like to = help > >> get > >> this fixed, I have one old i386 server I can try things out on if > >> that > >> helps and it's already on 9 stable and ZFS v28. > >> > > Btw, in previous discussions with Eric on this, he provided nfsstat > > output that seemed to indicate most of his RPC load from the Macs > > were Access and Getattr RPCs. > > > > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a > > significant part of this issue. I know nothing about ZFS, but I = believe > > it does always have ACLs enabled and presumably needs to check the > > ACL for each VOP_ACCESSX(). > > > > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and > > VOP_GETATTR() can look at these? >=20 > Indeed. However couldn't one simply disable ACL mode via; >=20 > zfs set aclinherit=3Ddiscard pool/dataset > zfs set aclmode=3Ddiscard pool/dataset >=20 > Eric, mind setting these and see? >=20 > Mid/late this week I'll be doing a rather large render farm test = amongst our Mac fleet against ZFS. >=20 > Will reply to this thread with outcome when I'm done. Should be = interesting. >=20 > - aurf >=20 > > > > rick > > > >> Thanks, > >> -- > >> Eric Browning > >> Systems Administrator > >> 801-984-7623 > >> > >> Skaggs Catholic Center > >> Juan Diego Catholic High School > >> Saint John the Baptist Middle > >> Saint John the Baptist Elementary > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" > >> > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >=20 >=20 >=20 >=20 > --=20 > Eric Browning > Systems Administrator > 801-984-7623 >=20 > Skaggs Catholic Center > Juan Diego Catholic High School > Saint John the Baptist Middle > Saint John the Baptist Elementary
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5969250F-0987-4304-BB95-52C7BAE8D84D>