Date: Wed, 15 Dec 1999 09:11:01 -0800 (PST) From: Matthew Dillon <dillon@apollo.backplane.com> To: Andrew Gallatin <gallatin@cs.duke.edu> Cc: freebsd-current@FreeBSD.ORG Subject: Re: Serious server-side NFS problem Message-ID: <199912151711.JAA32205@apollo.backplane.com> References: <14423.46117.353932.473968@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
:However, I'm seeing a showstopping problem when running newer kernels:
:When writing a large file via TCP, a Solaris 2.7 client pauses when
:closing the file, and appears to become stuck in an infinate loop.
:Eg:
:
:dd if=/dev/zero of=zot bs=64k count=8192
:8192+0 records in
:8192+0 records out
:^C <------------- wedge
:
:The process does not exit, and there is a flurry of activity between
:the client & server:
:
:solaris -> freebsd NFS C WRITE3 FH=F5CB at 369655808 for 32768 (ASYNC)
:________________________________
:freebsd -> solaris TCP D=843 S=2049 Ack=1906313520 Seq=94979688 Len=164 Win=33176
:freebsd -> solaris RPC R (#5146) XID=299504169 Success
:freebsd -> solaris NFS R WRITE3 OK 32768 (ASYNC)
:________________________________
This is very odd. Does it lockup with UDP or only with TCP? And only
with a solaris client?
It's possible that the problem may be related to changes in the TCP
stack rather then related to NFS. Looking at your trace output
I don't see anything wrong beyond the solaris client apparently repeating
write requests. Another possibility is that we somehow nixed the
commit rpc. All the repeated writes occur after the commit rpc. In
the protocol trace, though, the commit rpc succeeds. Very odd!
:I would think that this is not our fault, except things work just dandy
:with a kernel from July. In fact, the only way out of this situation
:is to reboot the FreeBSD NFS server into an older kernel.
:
:In the trace (about 30 seconds or so of the activity after dd
:finished, but before it exited) there are ~21,000 packets. There is a
:grand total of:
:
:NFS C WRITE3: 11024
:NFS R WRITE3 OK: 10499
:NFS C COMMIT3: 1
:NFS R COMMIT3 OK: 1
:
:In case more details are needed, I've left the complete trace in
:~gallatin/nfs-trace.gz on freefall.
I've looked at it -- it looks normal except for the repeated writes.
:Also, while read performance has improved by 44%, write performance
:has degraded by between 50 - 70% (FreeBSD clients)! Here are some
:quick benchmarks. Note that the file size of 512MB is larger than
:memory on both the server and client. Also note that the disk array
:on the server will read at 50MB/sec and write at 40MB/sec, so we are
:not disk bound ;-)
:
:- UDP NFS write performance from a FreeBSD client:
Ok, I'll take a look at it. I get 9 MBytes/sec writing over UDP links
but only with fast (400MHz class) clients. With a 200 MHz class client
I am only getting 4-5 MBytes/sec. Frankly, I should be getting 9 MB/sec
even with the slower client!
:UDP NFS Read performance has gotten better:
:
:July's kernel:
:% dd if=zot of=/dev/null bs=64k
:8192+0 records in
:8192+0 records out
:536870912 bytes transferred in 84.621477 secs (6344381 bytes/sec)
:
:Today's kernel:
:dd if=zot of=/dev/null bs=64k
:8192+0 records in
:8192+0 records out
:536870912 bytes transferred in 58.544409 secs (9170319 bytes/sec)
:
:Cheers,
:
:Drew
Yes, read performance has been improved in just the last few days
simply by adding a read heuristic to the server side for transfers off
the physical media.
-Matt
Matthew Dillon
<dillon@backplane.com>
:------------------------------------------------------------------------------
:Andrew Gallatin, Sr Systems Programmer http://www.cs.duke.edu/~gallatin
:Duke University Email: gallatin@cs.duke.edu
:Department of Computer Science Phone: (919) 660-6590
:
:
:
:To Unsubscribe: send mail to majordomo@FreeBSD.org
:with "unsubscribe freebsd-current" in the body of the message
:
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199912151711.JAA32205>