Date: Tue, 19 Nov 2013 11:38:33 -0800 From: aurfalien <aurfalien@gmail.com> To: Eric Browning <ericbrowning@skaggscatholiccenter.org> Cc: FreeBSD FS <freebsd-fs@freebsd.org> Subject: Re: Performance difference between UFS and ZFS with NFS Message-ID: <18391B9C-2FC4-427B-A4B6-1739B3C17498@gmail.com> In-Reply-To: <CAM=5oeBmCAq9unFGC2CBoJ3rZMm9MtDw1DWkFpo2ZqQtx3G%2B=Q@mail.gmail.com> References: <2103733116.16923158.1384866769683.JavaMail.root@uoguelph.ca> <9F76D61C-EFEB-44B3-9717-D0795789832D@gmail.com> <CAM=5oeAF2gfccrGNdbApUDpqRae4OQjZ7oaZZi4y1j%2BsF6PsTw@mail.gmail.com> <5969250F-0987-4304-BB95-52C7BAE8D84D@gmail.com> <CAM=5oeBmCAq9unFGC2CBoJ3rZMm9MtDw1DWkFpo2ZqQtx3G%2B=Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Wow, those are great mount options, I use em too :) Well, this is very interesting on the +3x access/getattrs with ZFS. I'll report back my findings as I'm going down a similar road, albeit = not home dirs but rendering using AE and C4D on many clients. Until then hoping some one chime in on this with some added nuggets. - aurf On Nov 19, 2013, at 11:11 AM, Eric Browning wrote: > Locking is set to locallocks, cache folders and similar folders are = redirected to the local hard drive. All applications run just fine = including Adobe CS6 and MS 2011 apps. >=20 > This is my client NFS conf: > nfs.client.mount.options =3D = noatime,nobrowse,tcp,vers=3D3,rsize=3D32768,wsize=3D32768,readahead=3D0,ac= regmax=3D3600,acdirmax=3D3600,locallocks,inet,noquota,nfc > nfs.client.statfs_rate_limit =3D 5 > nfs.client.access_for_getattr =3D 1 > nfs.client.is_mobile =3D 0 >=20 > I'm sure this is more complex than it needs to be and I can probably = get rid of most of this now, forcing nfc did cure some unicode issues = between mac and freebsd. Packets are not being fragmented and there are = only one or two errors here and there despite traversing vlans through = the core router, MSS is set at 1460. >=20 > One thing Rick M suggested is actually trying these entire setup on a = UFS system. I tested by copying my home folder to another server with a = UFS system and ran it for like 45 minutes and compared it to another 45 = minute jaunt on the main file server and I had about 3x less Access and = Getattrs on UFS than I had on ZFS. Seeing this prompted me to move one = server over to a UFS raid and since doing that it's like day and night = performance-wise.=20 >=20 > Server's NFS is set to 256 threads ARC is currently only at 46G of 56G = total and NFS is 9.9G on the ZFS server and CPU usage is 878%. On the = UFS server NFS is the same 256 threads and 9.9G but as I look at it with = currently 52 users logged in NFS is at CPU 0.00% usage. >=20 > This is the server NFS configs from rc.conf > ## NFS Server > rpcbind_enable=3D"YES" > nfs_server_enable=3D"YES" > mountd_flags=3D"-r -l" > nfsd_enable=3D"YES" > mountd_enable=3D"YES" > rpc_lockd_enable=3D"NO" > rpc_statd_enable=3D"NO" > nfs_server_flags=3D"-t -n 256" > nfsv4_server_enable=3D"NO" > nfsuserd_enable=3D"YES" >=20 > UFS Server mem stats: > Mem: 49M Active, 56G Inact, 3246M Wired, 1434M Cache, 1654M Buf, 1002M = Free > ARC: 1884K Total, 149K MFU, 1563K MRU, 16K Anon, 56K Header, 99K Other > Swap: 4096M Total, 528K Used, 4095M Free >=20 > ZFS mem stats: > Mem: 3180K Active, 114M Inact, 60G Wired, 1655M Buf, 2412M Free > ARC: 46G Total, 26G MFU, 13G MRU, 3099K Anon, 4394M Header, 4067M = Other > Swap: 4096M Total, 4096M Free >=20 >=20 >=20 > On Tue, Nov 19, 2013 at 11:25 AM, aurfalien <aurfalien@gmail.com> = wrote: > Curious. >=20 > Do you have NFS locking enabled client side? >=20 > Most likely you do as Mac Mail will not run w/o locks, nor will Adobe = prefs like temp cache. etc... >=20 > So being this is prolly the case, could it be a mem pressure issue and = not enough RAM? >=20 > So NFS locks take up RAM as does ARC. What are your mem stats and = swap stats during the 700% (yikes) experience? >=20 > - aurf >=20 > On Nov 19, 2013, at 10:19 AM, Eric Browning wrote: >=20 >> Aurf, >>=20 >> I ran those two commands and it doesn't seem to have made a = difference. Usage is still above 700% and it still takes 30s to list a = directory. The time to list is proportional to the number of users = logged in. On UFS with all students logged in and hammering away at = their files there is no noticeable speed decrease. >>=20 >>=20 >> On Tue, Nov 19, 2013 at 11:12 AM, aurfalien <aurfalien@gmail.com> = wrote: >>=20 >> On Nov 19, 2013, at 5:12 AM, Rick Macklem wrote: >>=20 >> > Eric Browning wrote: >> >> Some background: >> >> -Two identical servers, dual AMD Athlon 6220's 16 cores total @ = 3Ghz, >> >> -64GB ram each server >> >> -Four Intel DC S3700 800GB SSDs for primary storage, each server. >> >> -FreeBSD 9 stable as of 902503 >> >> -ZFS v28 and later updated to feature flags (v29?) >> >> -LSI 9200-8i controller >> >> -Intel I350T4 nic (only one port being used currently) using all = four >> >> in >> >> LACP overtaxed the server's NFS queue from what we found out = making >> >> the >> >> server basically unusable. >> >> >> >> There is definitely something going on between NFS and ZFS when = used >> >> as a >> >> file server (random workload) for mac home directories. They do = not >> >> jive >> >> well at all and pretty much drag down these beefy servers and = cause >> >> 20-30 >> >> second delays when just attempting to list a directory on Mac = 10.7, >> >> 10.8 >> >> clients although throughput seems fast when copying files. >> >> >> >> This server's NFS was sitting north of 700% (7+ cores) all day = long >> >> when >> >> using ZFSv28 raidz1. I have also tried stripe, compression on/off, >> >> sync >> >> enabled/disabled, and no dedup with 56GB of ram dedicated to ARC. >> >> I've >> >> tried just 100% stock settings in loader.conf and and some >> >> recommended >> >> tuning from various sources on the freebsd lists and other sites >> >> including >> >> the freebsd handbook. >> >> >> >> This is my mountpoint creation: >> >> zfs create -o mountpoint=3D/users -o sharenfs=3Don -o >> >> casesensitivity=3Dinsensitive -o aclmode=3Dpassthrough -o = compression=3Dlz4 >> >> -o >> >> atime=3Doff -o aclinherit=3Dpassthrough tank/users >> >> >> >> This last weekend I switched one of these servers over to a UFS = raid >> >> 0 >> >> setup and NFS now only eats about 36% of one core during the = initial >> >> login >> >> phase of 150-ish users over about 10 minutes and sits under 1-3% >> >> during >> >> normal usage and directories all list instantly even when drilling >> >> down 10 >> >> or so directories on the client's home files. The same NFS config = on >> >> server >> >> and clients are still active. >> >> >> >> Right now I'm going to have to abandon ZFS until it works with = NFS. >> >> I >> >> don't want to get into a finger pointing game, I'd just like to = help >> >> get >> >> this fixed, I have one old i386 server I can try things out on if >> >> that >> >> helps and it's already on 9 stable and ZFS v28. >> >> >> > Btw, in previous discussions with Eric on this, he provided nfsstat >> > output that seemed to indicate most of his RPC load from the Macs >> > were Access and Getattr RPCs. >> > >> > I suspect the way ZFS handles VOP_ACCESSX() and VOP_GETATTR() is a >> > significant part of this issue. I know nothing about ZFS, but I = believe >> > it does always have ACLs enabled and presumably needs to check the >> > ACL for each VOP_ACCESSX(). >> > >> > Hopefully someone familiar with how ZFS handles VOP_ACCESSX() and >> > VOP_GETATTR() can look at these? >>=20 >> Indeed. However couldn't one simply disable ACL mode via; >>=20 >> zfs set aclinherit=3Ddiscard pool/dataset >> zfs set aclmode=3Ddiscard pool/dataset >>=20 >> Eric, mind setting these and see? >>=20 >> Mid/late this week I'll be doing a rather large render farm test = amongst our Mac fleet against ZFS. >>=20 >> Will reply to this thread with outcome when I'm done. Should be = interesting. >>=20 >> - aurf >>=20 >> > >> > rick >> > >> >> Thanks, >> >> -- >> >> Eric Browning >> >> Systems Administrator >> >> 801-984-7623 >> >> >> >> Skaggs Catholic Center >> >> Juan Diego Catholic High School >> >> Saint John the Baptist Middle >> >> Saint John the Baptist Elementary >> >> _______________________________________________ >> >> freebsd-fs@freebsd.org mailing list >> >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> >> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >> >> >> > _______________________________________________ >> > freebsd-fs@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> > To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>=20 >>=20 >>=20 >>=20 >> --=20 >> Eric Browning >> Systems Administrator >> 801-984-7623 >>=20 >> Skaggs Catholic Center >> Juan Diego Catholic High School >> Saint John the Baptist Middle >> Saint John the Baptist Elementary >=20 >=20 >=20 >=20 > --=20 > Eric Browning > Systems Administrator > 801-984-7623 >=20 > Skaggs Catholic Center > Juan Diego Catholic High School > Saint John the Baptist Middle > Saint John the Baptist Elementary
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?18391B9C-2FC4-427B-A4B6-1739B3C17498>