Date: Fri, 08 Nov 2002 11:51:58 +0100 From: Andre Oppermann <oppermann@pipeline.ch> To: mallman@grc.nasa.gov Cc: freebsd-net@FreeBSD.ORG, Fran.Lawas-Grodek@grc.nasa.gov, Cindy.Tran@grc.nasa.gov, Joseph Ishac <Joseph.Ishac@grc.nasa.gov>, bdimond@grc.nasa.gov Subject: Re: Problem in High Speed and Long Delay with FreeBSD Message-ID: <3DCB974E.2703E90E@pipeline.ch> References: <200211080428.XAA29327@lawyers.ir.bbn.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Mark Allman wrote: > > Folks- > > Lots of interesting thoughts on this thread already. But, we have > not yet figured it out. So, a further data point... > > I have been playing this evening on my machine at home -- a way old > p5 running freebsd 4.7. I am seeing the same problem as we see at > GRC on the freebsd 4.1 boxes. As I see it we can have one of a few > possible problems: > > * For some reason the application is not able to fill the socket > buffer. > > * For some reason TCP is not grabbing everything from the socket > buffer. > > * TCP's congestion control is getting hosed somehow into thinking > we need to slow down. Try to enable the TCP inflight code from Matt Dillon. Have a look at man tuning and turn the sysctls on. Also turn on the debug sysctl and watch the bandwidthdelay product calculations. That might give you some more hints on what is going on. -- Andre > So, my idea goes something like this: Say I have a connection with a > X byte socket buffer. I start a timer, open a TCP connection, write > Y bytes (Y < X), close the connection and stop the timer. The time > should be very small in this case, because I am really timing how > long it take the app to send data to the OS. (I am not enabling > SO_LINGER). So, I ran some tests, as follows (sockbuf == X, dataen > == Y): > > sockbuf datalen snd_time rcv_time > --------------------------------------------- > 16384 15000 0.000 0.617 > 150000 140000 0.003 4.021 > 500000 495000 0.015 14.083 > 1000000 995000 0.042 28.577 > 1500000 1490000 0.079 47.986 > 1600000 1590000 0.088 44.055 > 1800000 1790000 0.108 50.810 > 1900000 1890000 0.117 55.010 > 2000000 1990000 1.011 57.666 > 2100000 2090000 3.845 60.233 > 3000000 2990000 39.594 122.308 > > So, except for the last 3 lines all the tests easily filled up the > socket buffers and exited. But, in the third line from the bottom > we started to see something else. We "filled" the buffers until we > could fill no more and had to wait for some ACKs to allow the buffer > to drain before dumping everything in it. But, the socket buffer is > bigger than the data stream so this should not be an issue, right? > The situation only gets worse as we increase the transfer sizes (and > socket buffer sizes). The break point seems to be somewhere near > 2MB. For instance, when sending almost 3MB the sender though the > transfer took ~40 seconds, while the receiver accurately shows ~122 > seconds. That shows that the sender had to wait for a third of the > data to drain from the network to get all its data into the socket > buffers. But, that should not have been. > > One more piece of information... The socket descriptor is placed in > non-blocking mode. I write in 8KB chunks. So, I do a select(), and > write accordingly. However, even though I cannot write into the > socket buffer at times, select() never fails to say that I can write > the descriptor *and* I never see a short write(). But, clearly I am > blocking or I'd shoot everything into the socket buffer. > > Does any of this make any sense to anyone? Any ideas on what might > be wrong here? Any suggestions on places to start looking? > > Thanks! > > allman > > -- > Mark Allman -- NASA GRC/BBN -- http://roland.grc.nasa.gov/~mallman/ > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-net" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3DCB974E.2703E90E>