Date: Thu, 13 Nov 2014 05:14:45 -0800 From: Mehmet Erol Sanliturk <m.e.sanliturk@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>, Mark Schouten <mark@tuxis.nl> Subject: Re: Slow nfsd write performance, tweaks needed Message-ID: <CAOgwaMsCybQv4D91xL=2r5o7b8GoAXPUyw76W23obVLq0h-v6A@mail.gmail.com> In-Reply-To: <1841043385.12560911.1415883615679.JavaMail.root@uoguelph.ca> References: <2826701214-10966@kerio.tuxis.nl> <1841043385.12560911.1415883615679.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Nov 13, 2014 at 5:00 AM, Rick Macklem <rmacklem@uoguelph.ca> wrote: > Mark Schouten wrote: > > Hi, > > > > > > I am in the process of switching from a ZFS On Linux-based NFS-server > > to a FreeBSD-based NFS-server. The FreeBSD implementation of ZFS is > > way superiour over ZoL, and the box serves as storage for a > > virtualizationplatform, so stability is welcome. :) > > > > > > The box is stable, but performs terribly. Surely, I'm doing something > > wrong, but I would like some tips and tricks to speed things up. > > > > > > > > > > Here's my setup: > > CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (HyperThreading is > > enabled) > > RAM: 64GB > > NIC: 2x igb in lagg0 (loadbalancing) > Oops, I didn't see this before my last post. igb had problems with > the 64K TSO issue and I'd try to get rid of lagg as well. > (You might be much better off just using a single net interface > without lagg.) > > Again, good luck with it, rick > > > Disks: > > > > export1 1.81T 914G 942G 49% 1.00x ONLINE - > > mirror 928G 457G 471G - > > da0 - - - - > > da1 - - - - > > mirror 928G 457G 471G - > > da2 - - - - > > da3 - - - - > > mirror 9.94G 173M 9.77G - > > da4p1 - - - - > > da5p1 - - - - > > cache - - - - - - > > da4p2 223G 223G 8M - > > da5p2 223G 223G 8M - > > > > > > da0-3 are 1TB WDs > > da4-5 are 240GB Samsung SSD 840s > > > > > > Here's (related) info from rc.conf. > > > > nfs_server_enable="YES" > > nfs_server_flags="-u -t -n 128" > > rpcbind_enable="YES" > > mountd_enable="YES" > > rpc_lockd_enable="YES" > > rpc_statd_enable="YES" > > > > > > > > > > I have compression enabled on all the ZFS-filesystems, and > > jumboframes are enabled on the nics. > > > > > > As soon as one of the (Linux) clients start to do some IO, NFS > > responsetimes go up bigtime (yesterday up to 13 seconds), while the > > hardware is pretty much idle, I must be doing something very wrong. > > I'm mostly a Linux-guy, so any hit with a FreeBSD cluebat is > > appreciated. > > > > > > Regards, > > > > > > -- > > Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ > > Mark Schouten | Tuxis Internet Engineering > > KvK: 61527076 | http://www.tuxis.nl/ > > T: 0318 200208 | info@tuxis.nl > A terrible write performance may come from Linux mount options in fstab : There is sync / async for the NFS connection : I was set sync : A file requiring 30 ( thirty ) SECONDs to write become 30 MINUTEs to write . After working to solve this problems by trying diffrerent parameter setting over many days , there only remained sync / async . When I selected async , writes turned to NORMAL . I do not know effect of compression ( my opinion is that it will not be "terrible" ) , but my suggestion is to check as a possible trouble point : sync / async in Linux computer ( in fstab or mount statement ) Thank you very much . Mehmet Erol Sanliturk
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOgwaMsCybQv4D91xL=2r5o7b8GoAXPUyw76W23obVLq0h-v6A>