Date: Sat, 8 Jan 2011 19:29:57 -0500 (EST) From: Rick Macklem <rmacklem@uoguelph.ca> To: Mark Saad <nonesuch@longcount.org> Cc: george+freebsd@mp5.com, freebsd-hackers@freebsd.org Subject: Re: NFS Performance Message-ID: <113335458.8956.1294532997422.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <AANLkTik7AAuqffjXjeT7L-hpXyXNkD-OjuDACJdd0kjd@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_8955_1251955688.1294532997410
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
> George
> I remember reading there was some sort of nfs issues in 8.1-RELEASE,
> a regression of some sort it was noted early on in the release. Have
> you tried this with 8.2-RC1 also what are your nfs client mount
> options ?
>
> On 1/8/11, george+freebsd@m5p.com <george+freebsd@m5p.com> wrote:
> > Among four machines on my network, I'm observing startling
> > differences
> > in NFS performance. All machines are AMD64, and rpc_statd,
> > rpc_lockd,
> > and amd are enabled on all four machines.
> >
> > wonderland:
> > hw.model: AMD Athlon(tm) II Dual-Core M32
> > hw.physmem: 293510758
> > ethernet: 100Mb/s
> > partition 1: FreeBSD 8.1-STABLE
> > partition 2: FreeBSD 7.3-STABLE
> >
> > scollay:
> > hw.model: AMD Sempron(tm) 140 Processor
> > hw.physmem: 186312294
> > ethernet: 1000Mb/s
> > FreeBSD 8.1-PRERELEASE
> >
> > sullivan:
> > hw.model: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+
> > hw.physmem: 4279980032
> > ethernet 1000Mb/s
> > FreeBSD 7.2-RELEASE
> >
> > mattapan:
> > hw.model: AMD Sempron(tm) Processor 2600+
> > hw.physmem: 456380416
> > ethernet: 1000Mb/s
> > FreeBSD 7.1-RELEASE
> >
> > Observed bytes per second (dd if=filename of=/dev/null bs=65536):
> > Source machine: mattapan scollay sullivan
> > Destination machine:
> > wonderland/7.3 870K 5.2M 1.8M
> > wonderland/8.1 496K 690K 420K
> > mattapan 38M 28M
> > scollay 33M 33M
> > sullivan 38M 5M
> >
> > There is one 10/100/1000Mb/s ethernet switch between the various
> > pairs
> > of machines.
> >
> > I'm startled by the numbers for wonderland, first because of how
> > much the
> > 100Mb/s interface slows things down, but even more because of how
> > much
> > difference there is on the identical hardware between FreeBSD 7 and
> > FreeBSD 8.
> >
> > Even more annoying when running 8.1 on wonderland, NFS simply locks
> > up
> > at random for roughly a minute and a half under high load (such as
> > when
> > firefox does a gazillion locked references to my places.sqlite
> > file),
> > leading to entertaining log message clusters such as:
> >
> > Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:17:47 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:17:47 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:18:01 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:18:01 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:18:08 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:18:08 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:20:21 wonderland last message repeated 2 times
> > Dec 29 08:20:22 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:20:22 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:20:36 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:20:36 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:21:05 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:21:10 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:22:22 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:22:22 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:22:24 wonderland last message repeated 2 times
> > Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:22:24 wonderland last message repeated 2 times
> > Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: is alive
> > again
> > Dec 29 08:22:27 wonderland kernel: nfs server home:/usr: not
> > responding
> > Dec 29 08:22:27 wonderland kernel: nfs server home:/usr: is alive
> > again
> >
> > The time stamps on the "not responding" messages are lies; the lack
> > of
> > response in each case was roughly a minute and a half earlier.
> > "home"
> > equals mattapan in the earlier table. During the periods of NFS lack
> > of response, mattapan is still completely functional, and a
> > concurrent
> > ssh session proceeds without difficulty. I can also make this happen
> > (though not as easily) on NFS from sullivan to wonderland, but I
> > haven't
> > seen it yet from scollay to wonderland. This problem never occurs
> > when
> > wonderland is running 7.3.
> >
> > Any suggestions? -- George Mitchell
> >
>
Please try the attached patch. (I don't read hackers, so I didn't see
this until to-day.) (I do read stable, current and fs.)
rick
ps: This patch is not even it head yet. The bug it fixes is in the client
side of the krpc, so it could affect the NFS client on any 8.x system.
------=_Part_8955_1251955688.1294532997410
Content-Type: text/x-patch; name=clntvc.patch
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=clntvc.patch
LS0tIHJwYy9jbG50X3ZjLmMuc2F2CTIwMTAtMTItMjkgMTk6MzY6MjAuMDAwMDAwMDAwIC0wNTAw
CisrKyBycGMvY2xudF92Yy5jCTIwMTAtMTItMjkgMjA6MDM6MjQuMDAwMDAwMDAwIC0wNTAwCkBA
IC05MTYsNiArOTE2LDExIEBACiAJCQkJbXR4X3VubG9jaygmY3QtPmN0X2xvY2spOwogCQkJCWJy
ZWFrOwogCQkJfQorCQkJaWYgKG0tPm1fbGVuIDwgc2l6ZW9mKHVpbnQzMl90KSkgeworCQkJCW0g
PSBtX3B1bGx1cChtLCBzaXplb2YodWludDMyX3QpKTsKKwkJCQlpZiAobSA9PSBOVUxMKQorCQkJ
CQlicmVhazsKKwkJCX0KIAkJCWJjb3B5KG10b2QobSwgdWludDMyX3QgKiksICZoZWFkZXIsIHNp
emVvZih1aW50MzJfdCkpOwogCQkJaGVhZGVyID0gbnRvaGwoaGVhZGVyKTsKIAkJCWN0LT5jdF9y
ZWNvcmQgPSBOVUxMOwo=
------=_Part_8955_1251955688.1294532997410--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?113335458.8956.1294532997422.JavaMail.root>