Date: Wed, 1 May 2013 21:29:44 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: "Marc G. Fournier" <scrappy@hub.org> Cc: freebsd-fs@freebsd.org Subject: Re: Initial NFS Test: Linux vs FreeBSD (769% slower) Message-ID: <971747745.58619.1367458184334.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <1032981589.58481.1367457568966.JavaMail.root@erie.cs.uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
I wrote: > Marc G. Fournier wrote: > > On 2013-04-30, at 05:29 , Rick Macklem < rmacklem@uoguelph.ca > > > wrote: > > > > > > Marc G. Fournier wrote: > > > > > > If you want, you can email me startup.pcap as an attachment and I'll > > take > > a look, but wireshark is pretty good at spotting TCP retransmits, > > etc. > > > > > > 'k, at 4.x Gig in size, doubt your mail server will handle me > > sending > > this to you :) Even compressed, it was going over 400M =E2=80=A6 I can = if > > you > > want it though =E2=80=A6 > > > > > > Baring that, if you want to give me pointers as to what I should be > > looking for? I have Wireshark installed / and the startup.pcap data > > loaded =E2=80=A6 the only thing that is jumping out at me is a bunch of > > lines > > where 'Length' is 32982 while most are 218 =E2=80=A6 highlighted in 'bl= ack > > background, red font' =E2=80=A6 for example: > > > > The big one is a write RPC and it will be a little more than 32768, > > if > > you've set wsize=3D32768. > > > > This can't be a capture for the "nfsstat" numbers you emailed the > > last > > time. > > (For one thing, that one didn't have any write RPCs counted.) Try > > and > > get > > a capture for the case where there are few NFS RPCs. (Did you > > capture > > for > > the first time doing the startup after doing a mount vs do an > > "nfsstat" > > for a subsequent startup?) Or, is this client doing something else > > on > > the > > network while the startup is happening? > > > > I may take a look at it, to see if I can spot anything weird, but a > > capture > > when it only does 88 RPCs is going to be much easier to look at. > > > > > > > > 'k, here is what I just ran now =E2=80=A6 igb1 is the private IP networ= k > > where > > the NFS mount is running from =E2=80=A6 host 192.168.1.5 is the host IP= of > > the > > server that I'm using for testing =E2=80=A6 and the results again have = WRITE > > calls in it =E2=80=A6 > > > > > > note that when I run the tcpdump command, I'm in /tmp on the local > > file system, so I'm not writing that log to the NFS server =E2=80=A6 an= d > > there > > is nothing else running on the NFS mount, since all tha tis on it is > > /usr/local/jboss=E2=80=A6 =E2=80=A6 the rest of /usr/local is on the lo= cal drives > > also > > =E2=80=A6 the WRITEs are to /usr/local/jboss/standalone/logs =E2=80=A6 = so not sure > > why > > RPC Counts for Writes is showing 0 change =E2=80=A6 > > > > > > I'm building a new bz2 file right now, but it looks pretty similar > > to > > the one I already sent you a URL for ... > > > > > > > > root@server04:/tmp # nfsstat -c; tcpdump -i igb1 -s 0 -w > > startup.pcap > > host 192.168.1.5 > > Client Info: > > Rpc Counts: > > Getattr Setattr Lookup Readlink Read Write Create Remove > > 2746536 821481 974263 18 2230948 2098303 160726 4954 > > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > > 1862 0 0 14724 950 16272 0 330261 > > Mknod Fsstat Fsinfo PathConf Commit > > 12 30926 6 0 0 > > Rpc Info: > > TimedOut Invalid X Replies Retries Requests > > 0 0 0 0 9432366 > > Cache Info: > > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses > > 26331524 2746535 20540473 974231 2380075 2225802 2618800 2097243 > > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses > > 1262 18 46863 15678 29941 0 22516391 330264 > > tcpdump: listening on igb1, link-type EN10MB (Ethernet), capture > > size > > 65535 bytes > > ^C5454815 packets captured > > 5458144 packets received by filter > > 0 packets dropped by kernel > > > > > > > > > > root@server04:/tmp # nfsstat -c > > Client Info: > > Rpc Counts: > > Getattr Setattr Lookup Readlink Read Write Create Remove > > 2746603 821481 974276 18 2230953 2098303 160726 4954 > > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access > > 1862 0 0 14724 950 16272 0 330275 > > Mknod Fsstat Fsinfo PathConf Commit > > 12 30926 6 0 0 > > Rpc Info: > > TimedOut Invalid X Replies Retries Requests > > 0 0 0 0 9432465 > > Cache Info: > > Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses > > 26332545 2746602 20540712 974244 2380801 2225807 2618800 2097243 > > BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses > > 1262 18 46863 15678 29941 0 22516700 330278 > > > > > > > > > > > > > > > > > > > > > > > > rick > > > > > > > > > > > > > > > > i am doing a bzip2 compressed file right now, and will make it > > available via HTTP, if you are interested =E2=80=A6 > > > > > > > > > > > > > > The only other thing I can suggest is taking the "soft,intr" options > > off > > your mount and see if that has any effect. Maybe some syscall is > > returning > > EINTR and confusing jboss? > > > > > > Tried removing soft,intr =E2=80=A6 no change, still around 240s =E2=80= =A6 > Well, I looked at the packet capture and, for some reason, it > repeatedly does a write of 1 byte to a file, followed by a read of > that file, over and over and ... again. I have no idea why the > app. does that. >=20 Oh, one more mount option you could try is "nocto". If the app. is repeatedly closing/opening the file, that might explain the repeated "write 1 byte; read some of the file"? rick > I also don't know why the "nfsstat -c" is bogus. Are you using "-t > oldnfs" > for the mount for these runs by any chance? If so, you need to use > "nfsstat -o -c". >=20 > rick > ps: To be honest, I doubt that anything can be done to speed this up, > except to move it to a local disk. >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?971747745.58619.1367458184334.JavaMail.root>