Date: Tue, 30 Mar 2010 20:20:08 GMT From: Bruce Evans <brde@optusnet.com.au> To: freebsd-bugs@FreeBSD.org Subject: Re: misc/145189: nfsd performs abysmally under load Message-ID: <201003302020.o2UKK8Fg011317@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR misc/145189; it has been noted by GNATS.
From: Bruce Evans <brde@optusnet.com.au>
To: Rich <rercola@acm.jhu.edu>
Cc: Bruce Evans <brde@optusnet.com.au>, freebsd-gnats-submit@freebsd.org,
freebsd-bugs@freebsd.org
Subject: Re: misc/145189: nfsd performs abysmally under load
Date: Wed, 31 Mar 2010 07:11:32 +1100 (EST)
This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.
--0-1912970806-1269979892=:2145
Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed
Content-Transfer-Encoding: QUOTED-PRINTABLE
On Tue, 30 Mar 2010, Rich wrote:
> On Tue, Mar 30, 2010 at 11:50 AM, Bruce Evans <brde@optusnet.com.au> wrot=
e:
>>> For instance, copying a 4GB file over NFSv3 from a ZFS filesystem with =
the
>>> following flags
>>> [rw,nosuid,hard,intr,nofsc,tcp,vers=3D3,rsize=3D8192,wsize=3D8192,slopp=
y,addr=3DX.X.X.X](Linux
>>> client, the above is the server), I achieve 2 MB/s, fluctuating between=
1
>>> and 3. (pv reports 2.23 MB/s avg)
I also tried various nfs r/w sizes and tcp/udp. The best sizes are
probably the fs block size or twice that (normally 16K for ffs). Old
versions of FreeBSD had even more bugs in this area and gave surprising
performance differences depending on the nfs r/w sizes or application
i/o sizes. In some cases smaller sizes worked best, apparently because
they avoided the stalls.
>>> ...
>> Enabling polling is a good way to destroy latency. =A0A ping latency of
>> ...
> Actually, we noticed that throughput appeared to get marginally better wh=
ile
> causing occasional bursts of crushing latency, but yes, we have it on in =
the
> kernel without using it in any actual NICs at present. :)
>
> But yes, I'm getting 40-90+ MB/s, occasionally slowing to 20-30 MB/s,
> average after copying a 6.5 GB file of 52.7 MB/s, on localhost IPv4,
> with no additional mount flags. {r,w}size=3D8192 on localhost goes up to
> 80-100 MB/s, with occasional sinks to 60 (average after copying
> another, separate 6.5 GB file: 77.3 MB/s).
I thought you said you often got 1-3MB/S.
> Also:
> 64 bytes from 127.0.0.1: icmp_seq=3D0 ttl=3D64 time=3D0.015 ms
> 64 bytes from 127.0.0.1: icmp_seq=3D1 ttl=3D64 time=3D0.049 ms
> 64 bytes from 127.0.0.1: icmp_seq=3D2 ttl=3D64 time=3D0.012 ms
Fairly normal slowness for -current.
> 64 bytes from [actual IP]: icmp_seq=3D0 ttl=3D64 time=3D0.019 ms
> 64 bytes from [actual IP]: icmp_seq=3D1 ttl=3D64 time=3D0.015 ms
Are these with external hardware NICs? Then 15 uS is excellent. Better
than I've ever seen. Very good hardware might be able to do this, but
I suspect it is for the local machine. BTW, I don't like the times
been reported in ms and sub-uS times not being supported. I sometimes
run Linux or cygwin ping and don't like it not supporting sub-mS times,
so that it always reports 0 for my average latency of 100 uS.
>> After various tuning and bug fixing (now partly committed by others) I g=
et
>> improvements like the following on low-end systems with ffs (I don't use
>> zfs):
>> - very low end with 100Mbps ethernet: little change; bulk transfers alwa=
ys
>> =A0went at near wire speed (about 10 MB/S)
>> - low end with 1Gbps/S: bulk transfers up from 20MB/S to 45MB/S (local f=
fs
>> =A050MB/S). =A0buildworld over nfs of 5.2 world down from 1200 seconds t=
o 800
>> =A0seconds (this one is very latency-sensitive. =A0Takes about 750 secon=
ds on
>> =A0local ffs).
>
> Is this on 9.0-CURRENT, or RELENG_8, or something else?
Mostly with 7-CURRENT or 8-CURRENT a couple of years ago. Sometimes with
a ~5.2-SERVER. nfs didn't vary much with the server, except there were
surprising differences due to latency that I never tracked down.
I forgot to mention another thing you can try easily:
- negative name caching. Improves latency. I used this to reduce makeworl=
d
times significantly, and it is now standard in -current but not
enabled by default.
Bruce
--0-1912970806-1269979892=:2145--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201003302020.o2UKK8Fg011317>