Date: Sun, 12 May 2013 08:48:03 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: "Marc G. Fournier" <scrappy@hub.org> Cc: freebsd-fs@freebsd.org Subject: Re: NFS Performance issue against NetApp Message-ID: <1966772823.291493.1368362883964.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <518F4130.6080201@hub.org>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_291492_1969524791.1368362883961 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Marc G. Fournier wrote: > 'k, here is on Linux ... this is right after rebooting the server, > doing > a mount and running the startup once: >=20 > Client rpc stats: > calls retrans authrefrsh > 40602 0 40609 >=20 > Client nfs v3: > null getattr setattr lookup access readlink > 0 0% 13000 32% 5 0% 6140 15% 6741 16% > 0 0% > read write create mkdir symlink mknod > 3556 8% 6711 16% 3743 9% 307 0% 0 0% > 0 0% > remove rmdir rename link readdir readdirplus > 1 0% 0 0% 0 0% 0 0% 16 0% > 380 0% > fsstat fsinfo pathconf commit > 0 0% 2 0% 1 0% 0 0% >=20 > One thing to note is that both Linux/FreeBSD have > "rsize=3D65536,wsize=3D65536" ... but there are 63x as many reads / 34x a= s > many writes on FreeBSD as on Linux ... ? >=20 > Just noticed this on the FreeBSD stats: >=20 > Rpc Info: > TimedOut Invalid X Replies Retries Requests > 0 0 0 0 818479 >=20 > 818k Retries? Is that normal ... ? >=20 > Also, the NetApp volumes being used here are not shared ... there are > no > other clients mounting these, and the Linux/FreeBSD volumes are > seperate > ... same size, same jboss install, same configuration, same war file > ... > I could mount /vol/linux_jboss onto the FreeBSD, or /vol/freebsd_jboss > onto the Linux, and they would load the same way ... in fact, the > jboss > install itself was done onto the FreeBSD and copied over to the Linux > ... and both are using OpenJDK7 ... I tried to make it as identical as > I > could ... >=20 >=20 > On 2013-05-11 7:27 PM, Marc G. Fournier wrote: > > > > With > > > > vfs.nfs.noconsist=3D3 ... 385595ms > > > > nfsstat -z before startup, nfsstat -c after: > > > > Client Info: > > Rpc Counts: > > Getattr Setattr Lookup Readlink Read Write Create > > Remove > > 332594 5 17238 0 224426 231137 > > 3743 1 > > Rename Link Symlink Mkdir Rmdir Readdir > > RdirPlus Access > > 0 0 0 307 0 71 0 8447 > > Mknod Fsstat Fsinfo PathConf Commit > > 0 509 0 0 0 > > Rpc Info: > > TimedOut Invalid X Replies Retries Requests > > 0 0 0 0 818479 > > Cache Info: > > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > > Hits Misses > > 608296 332596 526200 17245 -95425 224426 13178 > > 231137 > > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > > Hits Misses > > 0 0 1050 55 502 7 > > 543340 8448 > > Ok, so disabling the mtime based cache consistency doesn't make much difference. Forget about that one. I've attached another patch (which you probably shouldn't use for a production system either) to be tried instead of the last one. (This one is basically "work in progress" by Alexander Kabaev for better performance during file linking. I hope he doesn't mind me posting it.) rick > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > vfs.nfs.noconsist=3D2 ... 392201ms > > > > Client Info: > > Rpc Counts: > > Getattr Setattr Lookup Readlink Read Write Create > > Remove > > 332557 5 17228 0 224421 231131 > > 3743 1 > > Rename Link Symlink Mkdir Rmdir Readdir > > RdirPlus Access > > 0 0 0 307 0 72 0 8430 > > Mknod Fsstat Fsinfo PathConf Commit > > 0 502 0 0 0 > > Rpc Info: > > TimedOut Invalid X Replies Retries Requests > > 0 0 0 0 818395 > > Cache Info: > > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > > Hits Misses > > 607834 332557 525801 17231 -95401 224421 13178 > > 231131 > > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > > Hits Misses > > 0 0 1028 56 502 0 > > 542925 8431 > > > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > vfs.nfs.noconsist=3D0 ... 391622ms > > > > > > Client Info: > > Rpc Counts: > > Getattr Setattr Lookup Readlink Read Write Create > > Remove > > 236122 5 17221 0 230575 230823 > > 3743 1 > > Rename Link Symlink Mkdir Rmdir Readdir > > RdirPlus Access > > 0 0 0 307 0 71 0 8425 > > Mknod Fsstat Fsinfo PathConf Commit > > 0 516 0 0 0 > > Rpc Info: > > TimedOut Invalid X Replies Retries Requests > > 0 0 0 0 727799 > > Cache Info: > > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW > > Hits Misses > > 711860 236124 526549 17225 -101525 230490 13178 > > 230823 > > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs > > Hits Misses > > 0 0 1057 55 516 0 > > 543709 8425 > > > > > > I checked a second time with nonconsist=3D0, and the nfsstat -c values > > seem to come out pretty much the same ... > > > > I'm going to head down to the office and try again with Solaris (I'd > > have to re-install, since I used that system for the Solaris), and > > see > > what nfsstat -c results I get out of that ... will post a followup > > on > > this when completed ... > > > > > > > > On 2013-05-10 5:32 PM, Rick Macklem wrote: > >> Marc G. Fournier wrote: > >>> FYI =E2=80=A6 I just installed Solaris 11 onto the same hardware and = ran > >>> the > >>> same test =E2=80=A6 so far, I'm seeing: > >>> > >>> Linux @ ~30s > >>> Solaris @ ~44s > >>> > >>> OpenBSD @ ~200s > >>> FreeBSD @ ~240s > >>> > >>> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' > >>> issue > >>> =E2=80=A6 same as 9.x =E2=80=A6 I could see Linux 'cutting corners', = but > >>> Oracle/Solaris too =E2=80=A6 ? > >>> > >> The three client implementations (BSD, Linux, Solaris) were > >> developed > >> independently and, as such, will all implement somewaht different > >> caching algorithms (the RFCs specify what goes on the wire, but say > >> little w.r.t. client side caching). > >> > >> I have a attached a patch that might be useful for determining if > >> the client side buffer cache consistency algorithm in FreeBSD is > >> causing the slow startup of jboss. Do not run this patch on a > >> production system, since it pretty well disables all buffer cache > >> coherency (ie. if another client modifies a file, the patched > >> client > >> won't notice and will continue to cache stale file data). > >> > >> If the patch does speed up startup of jboss significantly, you can > >> use the sysctl: > >> vfs.nfs.noconsist > >> to check for which coherency check is involved by decreasing the > >> value for the sysctl by 1 and then trying a startup again. (When > >> vfs.nfs.noconsist=3D0, normal cache coherency will be applied.) > >> > >> I have no idea if buffer cache coherency is a factor, but trying > >> the attached patch might determine if it is. > >> > >> Note that you have never posted updated "nfsstat -c" values. > >> (Remember that what you posted indicated 88 RPCs, which seemed > >> bogus.) Finding out if FreeBSD does a lot more of certain RPCs > >> that Linux/Solaris might help isolate what is going on. > >> > >> rick > >> > >>> On 2013-05-03, at 04:50 , Mark Felder <feld@feld.me> wrote: > >>> > >>>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier > >>>> <scrappy@hub.org> wrote: > >>>> > >>>>> Hadn't thought to do so with Linux, but =E2=80=A6 > >>>>> Linux =E2=80=A6=E2=80=A6. 20732ms, 20117ms, 20935ms, 20130ms, 20560= ms > >>>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms > >>>> Please make sure both platforms are using similar atime settings. > >>>> I > >>>> think most distros use ext4 with diratime by default. I'd just do > >>>> noatime on both platforms to be safe. > >>>> _______________________________________________ > >>>> freebsd-fs@freebsd.org mailing list > >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>>> To unsubscribe, send any mail to > >>>> "freebsd-fs-unsubscribe@freebsd.org" > >>> _______________________________________________ > >>> freebsd-fs@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >>> To unsubscribe, send any mail to > >>> "freebsd-fs-unsubscribe@freebsd.org" > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to > > "freebsd-fs-unsubscribe@freebsd.org" ------=_Part_291492_1969524791.1368362883961 Content-Type: text/x-patch; name=nfs.diff Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=nfs.diff ZGlmZiAtLWdpdCBhL3N5cy9mcy9uZnNjbGllbnQvbmZzX2NsYmlvLmMgYi9zeXMvZnMvbmZzY2xp ZW50L25mc19jbGJpby5jDQppbmRleCBhMGVjOGVlLi5lNmE3MjY3IDEwMDY0NA0KLS0tIGEvc3lz L2ZzL25mc2NsaWVudC9uZnNfY2xiaW8uYw0KKysrIGIvc3lzL2ZzL25mc2NsaWVudC9uZnNfY2xi aW8uYw0KQEAgLTEwMzEsMTMgKzEwMzEsMTYgQEAgZmx1c2hfYW5kX3Jlc3RhcnQ6DQogCQlsYm4g PSB1aW8tPnVpb19vZmZzZXQgLyBiaW9zaXplOw0KIAkJb24gPSB1aW8tPnVpb19vZmZzZXQgLSAo bGJuICogYmlvc2l6ZSk7DQogCQluID0gTUlOKCh1bnNpZ25lZCkoYmlvc2l6ZSAtIG9uKSwgdWlv LT51aW9fcmVzaWQpOw0KKyNpZiAwDQogYWdhaW46DQorI2VuZGlmDQogCQkvKg0KIAkJICogSGFu ZGxlIGRpcmVjdCBhcHBlbmQgYW5kIGZpbGUgZXh0ZW5zaW9uIGNhc2VzLCBjYWxjdWxhdGUNCiAJ CSAqIHVuYWxpZ25lZCBidWZmZXIgc2l6ZS4NCiAJCSAqLw0KIAkJbXR4X2xvY2soJm5wLT5uX210 eCk7DQotCQlpZiAodWlvLT51aW9fb2Zmc2V0ID09IG5wLT5uX3NpemUgJiYgbikgew0KKwkJaWYg KGxibiA9PSAobnAtPm5fc2l6ZSAvIGJpb3NpemUpICYmDQorCQkgICAgdWlvLT51aW9fb2Zmc2V0 ICsgbiA+IG5wLT5uX3NpemUgJiYgbikgew0KIAkJCW10eF91bmxvY2soJm5wLT5uX210eCk7DQog CQkJLyoNCiAJCQkgKiBHZXQgdGhlIGJ1ZmZlciAoaW4gaXRzIHByZS1hcHBlbmQgc3RhdGUgdG8g bWFpbnRhaW4NCkBAIC0xMDQ1LDcgKzEwNDgsNyBAQCBhZ2FpbjoNCiAJCQkgKiBuZnNub2RlIGFm dGVyIHdlIGhhdmUgbG9ja2VkIHRoZSBidWZmZXIgdG8gcHJldmVudA0KIAkJCSAqIHJlYWRlcnMg ZnJvbSByZWFkaW5nIGdhcmJhZ2UuDQogCQkJICovDQotCQkJYmNvdW50ID0gb247DQorCQkJYmNv dW50ID0gbnAtPm5fc2l6ZSAtIChsYm4gKiBiaW9zaXplKTsNCiAJCQlicCA9IG5mc19nZXRjYWNo ZWJsayh2cCwgbGJuLCBiY291bnQsIHRkKTsNCiANCiAJCQlpZiAoYnAgIT0gTlVMTCkgew0KQEAg LTEwNTgsNyArMTA2MSw3IEBAIGFnYWluOg0KIAkJCQltdHhfdW5sb2NrKCZucC0+bl9tdHgpOw0K IA0KIAkJCQlzYXZlID0gYnAtPmJfZmxhZ3MgJiBCX0NBQ0hFOw0KLQkJCQliY291bnQgKz0gbjsN CisJCQkJYmNvdW50ID0gb24gKyBuOw0KIAkJCQlhbGxvY2J1ZihicCwgYmNvdW50KTsNCiAJCQkJ YnAtPmJfZmxhZ3MgfD0gc2F2ZTsNCiAJCQl9DQpAQCAtMTE1NCw2ICsxMTU3LDcgQEAgYWdhaW46 DQogCQlpZiAoYnAtPmJfZGlydHlvZmYgPj0gYnAtPmJfZGlydHllbmQpDQogCQkJYnAtPmJfZGly dHlvZmYgPSBicC0+Yl9kaXJ0eWVuZCA9IDA7DQogDQorI2lmIDANCiAJCS8qDQogCQkgKiBJZiB0 aGUgbmV3IHdyaXRlIHdpbGwgbGVhdmUgYSBjb250aWd1b3VzIGRpcnR5DQogCQkgKiBhcmVhLCBq dXN0IHVwZGF0ZSB0aGUgYl9kaXJ0eW9mZiBhbmQgYl9kaXJ0eWVuZCwNCkBAIC0xMTc5LDYgKzEx ODMsMTQgQEAgYWdhaW46DQogCQkJfQ0KIAkJCWdvdG8gYWdhaW47DQogCQl9DQorI2Vsc2UNCisJ CS8qDQorCQkgKiBSZWxheCBjb2hlcmVuY3kgYSBiaXQgZm9yIHRoZSBzYWtlIG9mIHBlcmZvcm1h bmNlIGFuZA0KKwkJICogZXhwYW5kIHRoZSBjdXJyZW50IGRpcnR5IHJlZ2lvbiB0byBjb250YWlu IHRoZSBuZXcNCisJCSAqIHdyaXRlIGV2ZW4gaWYgaXQgbWVhbnMgd2UgbWFyayBzb21lIG5vbi1k aXJ0eSBkYXRhIGFzDQorCQkgKiBkaXJ0eS4gIFRoaXMgc2hvdWxkIHByb2JhYmx5IGJlIGNvbmZp Z3VyYWJsZS4NCisJCSAqLw0KKyNlbmRpZg0KIA0KIAkJbG9jYWxfcmVzaWQgPSB1aW8tPnVpb19y ZXNpZDsNCiAJCWVycm9yID0gdm5faW9fZmF1bHRfdWlvbW92ZSgoY2hhciAqKWJwLT5iX2RhdGEg KyBvbiwgbiwgdWlvKTsNCmRpZmYgLS1naXQgYS9zeXMvbmZzY2xpZW50L25mc19iaW8uYyBiL3N5 cy9uZnNjbGllbnQvbmZzX2Jpby5jDQppbmRleCA2MzBhN2ZmLi45ZDhkYzdjIDEwMDY0NA0KLS0t IGEvc3lzL25mc2NsaWVudC9uZnNfYmlvLmMNCisrKyBiL3N5cy9uZnNjbGllbnQvbmZzX2Jpby5j DQpAQCAtMTEzMyw2ICsxMTMzLDcgQEAgYWdhaW46DQogCQlpZiAoYnAtPmJfZGlydHlvZmYgPj0g YnAtPmJfZGlydHllbmQpDQogCQkJYnAtPmJfZGlydHlvZmYgPSBicC0+Yl9kaXJ0eWVuZCA9IDA7 DQogDQorI2lmIDANCiAJCS8qDQogCQkgKiBJZiB0aGUgbmV3IHdyaXRlIHdpbGwgbGVhdmUgYSBj b250aWd1b3VzIGRpcnR5DQogCQkgKiBhcmVhLCBqdXN0IHVwZGF0ZSB0aGUgYl9kaXJ0eW9mZiBh bmQgYl9kaXJ0eWVuZCwNCkBAIC0xMTU4LDYgKzExNTksMTQgQEAgYWdhaW46DQogCQkJfQ0KIAkJ CWdvdG8gYWdhaW47DQogCQl9DQorI2Vsc2UNCisJCS8qDQorCQkgKiBSZWxheCBjb2hlcmVuY3kg YSBiaXQgZm9yIHRoZSBzYWtlIG9mIHBlcmZvcm1hbmNlIGFuZA0KKwkJICogZXhwYW5kIHRoZSBj dXJyZW50IGRpcnR5IHJlZ2lvbiB0byBjb250YWluIHRoZSBuZXcNCisJCSAqIHdyaXRlIGV2ZW4g aWYgaXQgbWVhbnMgd2UgbWFyayBzb21lIG5vbi1kaXJ0eSBkYXRhIGFzDQorCQkgKiBkaXJ0eS4g IFRoaXMgc2hvdWxkIHByb2JhYmx5IGJlIGNvbmZpZ3VyYWJsZS4NCisJCSAqLw0KKyNlbmRpZg0K IA0KIAkJZXJyb3IgPSB1aW9tb3ZlKChjaGFyICopYnAtPmJfZGF0YSArIG9uLCBuLCB1aW8pOw0K IA0K ------=_Part_291492_1969524791.1368362883961--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1966772823.291493.1368362883964.JavaMail.root>