Date: Wed, 24 Jul 2002 15:52:04 -0500 From: "Jaime Bozza" <jbozza@thinkburst.com> To: "'Matthew Dillon'" <dillon@apollo.backplane.com> Cc: <stable@FreeBSD.ORG>, "'Olaf R'" <olaf@keghouse.net> Subject: RE: RE: Abominable NFSv3 read performance / FreeBSD server / Solaris client Message-ID: <025c01c23353$f75cc730$6401010a@bozza> In-Reply-To: <200207241715.g6OHF9Ne028675@apollo.backplane.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Ok, I fiddled with the tcp_recv_hiwat parameter on Solaris 8, along
with the tcp.sendspace parameter on FreeBSD. No matter what, anything
above 32767 would continually get the delays.
Just another FYI, by default, Solaris 8 sets tcp_recv_hiwat to 24576.
And another interesting note. Keeping all defaults (sendspace=32768,
recv_hiwat=24576) but setting rsize to 28672 or 30720, I get almost a
30% increase in speed (for this particular test) from rsize=20480.
(13 seconds or so instead of 18 or so)
In typing this message and testing at the same time, I came across these
two parameters:
nfs:nfs3_bsize
nfs:nfs3_max_transfer_size
Unfortunately, I don't think I can change them with a running system.
(Have to set them in /etc/system and reboot to make the changes)
But regardless, both are set to 32768. I wouldn't be surprised if
rsize=32768
would work fine if I increased both of these values. Here are the
beginnings
of a couple of tcpdumps.
FreeBSD to FreeBSD:
14:52:36.435205 freebsd-client.1700858811 > freebsd-server.nfs: 132
lookup [|nfs]
14:52:36.435334 freebsd-server.nfs > freebsd-client.1700858811: reply ok
236 lookup [|nfs]
14:52:36.435580 freebsd-client.1700858812 > freebsd-server.nfs: 132
lookup [|nfs]
14:52:36.435625 freebsd-server.nfs > freebsd-client.1700858812: reply ok
236 lookup [|nfs]
14:52:36.435844 freebsd-client.1700858813 > freebsd-server.nfs: 144
lookup [|nfs]
14:52:36.435884 freebsd-server.nfs > freebsd-client.1700858813: reply ok
236 lookup [|nfs]
14:52:36.436104 freebsd-client.1700858814 > freebsd-server.nfs: 124
access [|nfs]
14:52:36.436140 freebsd-server.nfs > freebsd-client.1700858814: reply ok
120 access c 65610bbf
14:52:36.436326 freebsd-client.1700858815 > freebsd-server.nfs: 124
access [|nfs]
14:52:36.436356 freebsd-server.nfs > freebsd-client.1700858815: reply ok
120 access c 65610bc0
14:52:36.441686 freebsd-client.1700858816 > freebsd-server.nfs: 132 read
[|nfs]
14:52:36.441727 freebsd-client.1700858817 > freebsd-server.nfs: 132 read
[|nfs]
14:52:36.441845 freebsd-server.nfs > freebsd-client.1700858816: reply ok
1472 read (frag 42077:1480@0+)
14:52:36.441851 freebsd-server > freebsd-client: (frag 42077:1480@1480+)
14:52:36.441858 freebsd-server > freebsd-client: (frag 42077:1480@2960+)
14:52:36.441863 freebsd-server > freebsd-client: (frag 42077:1480@4440+)
14:52:36.441870 freebsd-server > freebsd-client: (frag 42077:1480@5920+)
14:52:36.441876 freebsd-server > freebsd-client: (frag 42077:928@7400)
14:52:36.441965 freebsd-server.nfs > freebsd-client.1700858817: reply ok
1472 read (frag 42078:1480@0+)
14:52:36.441971 freebsd-server > freebsd-client: (frag 42078:1480@1480+)
14:52:36.441976 freebsd-server > freebsd-client: (frag 42078:1480@2960+)
14:52:36.441981 freebsd-server > freebsd-client: (frag 42078:1480@4440+)
14:52:36.441986 freebsd-server > freebsd-client: (frag 42078:1480@5920+)
14:52:36.441991 freebsd-server > freebsd-client: (frag 42078:928@7400)
14:52:36.443058 freebsd-client.1700858818 > freebsd-server.nfs: 132 read
[|nfs]
FreeBSD to Solaris:
14:53:22.433685 solaris-client.1823430800 > freebsd-server.nfs: 172
lookup [|nfs] (DF)
14:53:22.433820 freebsd-server.nfs > solaris-client.1823430800: reply ok
240 lookup [|nfs] (DF)
14:53:22.434593 solaris-client.1823430801 > freebsd-server.nfs: 172
lookup [|nfs] (DF)
14:53:22.434647 freebsd-server.nfs > solaris-client.1823430801: reply ok
240 lookup [|nfs] (DF)
14:53:22.435247 solaris-client.1823430802 > freebsd-server.nfs: 184
lookup [|nfs] (DF)
14:53:22.435292 freebsd-server.nfs > solaris-client.1823430802: reply ok
240 lookup [|nfs] (DF)
14:53:22.436768 solaris-client.1823430803 > freebsd-server.nfs: 164
access [|nfs] (DF)
14:53:22.436805 freebsd-server.nfs > solaris-client.1823430803: reply ok
124 access c 6caf5894 (DF)
14:53:22.440026 solaris-client.1823430804 > freebsd-server.nfs: 172 read
[|nfs] (DF)
14:53:22.440120 solaris-client.1823430805 > freebsd-server.nfs: 172 read
[|nfs] (DF)
14:53:22.440142 freebsd-server.nfsd > solaris-client.962: . ack
3598640967 win 33176 (DF)
14:53:22.440267 freebsd-server.nfs > solaris-client.1823430804: reply ok
984 read (DF)
14:53:22.440276 freebsd-server.nfs > solaris-client.2661457985: reply
ERR 984 (DF)
14:53:22.440284 freebsd-server.nfs > solaris-client.808722359: reply ERR
984 (DF)
14:53:22.440293 freebsd-server.nfs > solaris-client.377809557: reply ERR
984 (DF)
14:53:22.440303 freebsd-server.nfs > solaris-client.3567562942: reply
ERR 984 (DF)
14:53:22.440313 freebsd-server.nfs > solaris-client.2021609926: reply
ERR 984 (DF)
14:53:22.440323 freebsd-server.nfs > solaris-client.1352417867: reply
ERR 984 (DF)
14:53:22.440331 freebsd-server.nfs > solaris-client.2579146601: reply
ERR 984 (DF)
14:53:22.440341 freebsd-server.nfs > solaris-client.1630463784: reply
ERR 984 (DF)
14:53:22.440351 freebsd-server.nfs > solaris-client.1567037614: reply
ERR 984 (DF)
14:53:22.440360 freebsd-server.nfs > solaris-client.1074370493: reply
ERR 984 (DF)
14:53:22.440368 freebsd-server.nfs > solaris-client.3968313526: reply
ERR 984 (DF)
14:53:22.440377 freebsd-server.nfs > solaris-client.1204350401: reply
ERR 984 (DF)
14:53:22.440386 freebsd-server.nfs > solaris-client.1850210752: reply
ERR 984 (DF)
14:53:22.440396 freebsd-server.nfs > solaris-client.787132493: reply ERR
984 (DF)
14:53:22.440405 freebsd-server.nfs > solaris-client.2182931510: reply
ERR 984 (DF)
14:53:22.440413 freebsd-server.nfs > solaris-client.2865285158: reply
ERR 984 (DF)
14:53:22.440422 freebsd-server.nfs > solaris-client.717978252: reply ERR
984 (DF)
14:53:22.440431 freebsd-server.nfs > solaris-client.2724501050: reply
ERR 984 (DF)
14:53:22.440441 freebsd-server.nfs > solaris-client.581303660: reply ERR
984 (DF)
14:53:22.440452 freebsd-server.nfs > solaris-client.659913908: reply ERR
984 (DF)
14:53:22.440459 freebsd-server.nfs > solaris-client.50598912: reply ERR
984 (DF)
14:53:22.440468 freebsd-server.nfs > solaris-client.3323815642: reply
ERR 984 (DF)
14:53:22.440475 freebsd-server.nfs > solaris-client.1853246635: reply
ERR 984 (DF)
14:53:22.440484 freebsd-server.nfs > solaris-client.2530310829: reply
ERR 984 (DF)
I'm not really good at interpreting what I'm seeing here, but I know
that the first
couple of lines are looking up the file. After that, I'm at a loss with
the Solaris
dump, other than the fact that I'm pretty sure the "reply ERR" are
causing delays
somewhere. (The FreeBSD one is so clean!)
Does FreeBSD attempt to send packets out using a size of NFS_MAXPACKET
(33172?) all
the time? Could the "win 33176" be centered around that? From what I'm
seeing
(correct me if I'm wrong), FreeBSD is sending 8328 bytes in each TCP
packet when
Sending to a FreeBSD client.
I don't know enough about the internals of tcp on both systems to really
figure
out what's really going on here. Any pointers on where to research next
would be
helpful. I'm pretty sure that it would have to be something on Solaris
that's
causing the problem, but I'm at a loss as to what.
Jaime
-----Original Message-----
From: Matthew Dillon [mailto:dillon@apollo.backplane.com]
Sent: Wednesday, July 24, 2002 12:15 PM
To: Jaime Bozza
Cc: stable@FreeBSD.ORG; 'Olaf R'
Subject: Re: RE: Abominable NFSv3 read performance / FreeBSD server /
Solaris client
Increase sendspace on the server and recvspace on the client, and
make
sure they've actually increased. A tcpdump should show much larger
window
advertisements from the solaris client, and netstat -tn on the
FreeBSD box
during the transfer (over and over again) should show that the
FreeBSD box
is able to queue up more then 32K.
-Matt
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?025c01c23353$f75cc730$6401010a>