Date: Thu, 07 Nov 2002 23:28:33 -0500 From: Mark Allman <mallman@grc.nasa.gov> To: freebsd-net@FreeBSD.ORG Cc: "Joseph Ishac" <Joseph.Ishac@grc.nasa.gov>, bdimond@grc.nasa.gov Subject: Re: Problem in High Speed and Long Delay with FreeBSD Message-ID: <200211080428.XAA29327@lawyers.ir.bbn.com> In-Reply-To: <20021101112001.E3259@grc.nasa.gov>
next in thread | previous in thread | raw e-mail | index | archive | help
Folks-
Lots of interesting thoughts on this thread already. But, we have
not yet figured it out. So, a further data point...
I have been playing this evening on my machine at home -- a way old
p5 running freebsd 4.7. I am seeing the same problem as we see at
GRC on the freebsd 4.1 boxes. As I see it we can have one of a few
possible problems:
* For some reason the application is not able to fill the socket
buffer.
* For some reason TCP is not grabbing everything from the socket
buffer.
* TCP's congestion control is getting hosed somehow into thinking
we need to slow down.
So, my idea goes something like this: Say I have a connection with a
X byte socket buffer. I start a timer, open a TCP connection, write
Y bytes (Y < X), close the connection and stop the timer. The time
should be very small in this case, because I am really timing how
long it take the app to send data to the OS. (I am not enabling
SO_LINGER). So, I ran some tests, as follows (sockbuf == X, dataen
== Y):
sockbuf datalen snd_time rcv_time
---------------------------------------------
16384 15000 0.000 0.617
150000 140000 0.003 4.021
500000 495000 0.015 14.083
1000000 995000 0.042 28.577
1500000 1490000 0.079 47.986
1600000 1590000 0.088 44.055
1800000 1790000 0.108 50.810
1900000 1890000 0.117 55.010
2000000 1990000 1.011 57.666
2100000 2090000 3.845 60.233
3000000 2990000 39.594 122.308
So, except for the last 3 lines all the tests easily filled up the
socket buffers and exited. But, in the third line from the bottom
we started to see something else. We "filled" the buffers until we
could fill no more and had to wait for some ACKs to allow the buffer
to drain before dumping everything in it. But, the socket buffer is
bigger than the data stream so this should not be an issue, right?
The situation only gets worse as we increase the transfer sizes (and
socket buffer sizes). The break point seems to be somewhere near
2MB. For instance, when sending almost 3MB the sender though the
transfer took ~40 seconds, while the receiver accurately shows ~122
seconds. That shows that the sender had to wait for a third of the
data to drain from the network to get all its data into the socket
buffers. But, that should not have been.
One more piece of information... The socket descriptor is placed in
non-blocking mode. I write in 8KB chunks. So, I do a select(), and
write accordingly. However, even though I cannot write into the
socket buffer at times, select() never fails to say that I can write
the descriptor *and* I never see a short write(). But, clearly I am
blocking or I'd shoot everything into the socket buffer.
Does any of this make any sense to anyone? Any ideas on what might
be wrong here? Any suggestions on places to start looking?
Thanks!
allman
--
Mark Allman -- NASA GRC/BBN -- http://roland.grc.nasa.gov/~mallman/
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200211080428.XAA29327>