Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 18 Nov 2006 22:18:46 -0500
From:      Bill Moran <wmoran@collaborativefusion.com>
To:        freebsd@sopwith.solgatos.com
Cc:        freebsd-questions@freebsd.org
Subject:   Re: TCP parameters and interpreting tcpdump output
Message-ID:  <20061118221846.3ae8c1ee.wmoran@collaborativefusion.com>
In-Reply-To: <200611182020.UAA03897@sopwith.solgatos.com>
References:  <200611182020.UAA03897@sopwith.solgatos.com>

next in thread | previous in thread | raw e-mail | index | archive | help

My comments are based both on the packet dump here and the source code
you posted earlier ...

On Sat, 18 Nov 2006 12:20:33 +0000
Dieter <freebsd@sopwith.solgatos.com> wrote:
> In the tcpdump output below, the src machine is sending data to the
> bsd machine.  At one point during this test, the bsd machine is slowly
> falling behind, as shown in the smaller and smaller window size.
> 
> It looks like at one point, the bsd machine takes 5.5 seconds to
> ack a packet.  :-(  Am I interpreting the -ttt delta time correctly?

I think so.  My guess would be that your process blocked on stdout.
You don't mention what you're doing with stdout from the program, are
you just letting it scroll on the terminal, or redirecting it to a file?

In any case, if stdout blocks (because the disk is slow, for example)
your recv buffer will fill up until your program can get the data
written.  As an experiment, try running the process and redirecting
stdout to /dev/null -- if it doesn't exhibit the problem, then you
need to look at where you're actually storing the data and speed that
part up.

Is the data coming in at a fairly constant rate?  If not, and you're only
having trouble when it bursts, you can work around this by keeping a
ring buffer and doing non-blocking writes.  It'll make your code more
complex, but it will allow your program to absorb some of the bursting
data and keep the window from closing -- especially if you've got plenty
of RAM.  If this is a constant rate of data, however, you're going to
need a faster way to store or process or whatever you do with it as it
comes in.

> Getsockopt() says bsd machine's send buffer = 33580, rec buffer = 197100
> Is there a way for the bsd machine to find out what the src machine's
> send buffer size is?

I don't know of any way but to ask them.  Again, it will only help if
it's a bursting problem.  If your receiving process is unable to keep
up with the required throughput, eventually you'll block and the other
end will run out of buffer space.

> I doubt that it is large enough for 5.5 seconds'
> worth of data, but it would be nice to know what the goal is.
> 
> Towards the end of the log, it looks to me like both sides are a
> bit quick to resend data and acks?
> 
> 000016 IP bsd.12340 > src.rfe: . ack 1201508 win 65535
> 000641 IP src.rfe > bsd.12340: P 1201508:1202824(1316) ack 1 win 4096
> 000780 IP src.rfe > bsd.12340: P 1202824:1204140(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1204140 win 65535
> 000953 IP src.rfe > bsd.12340: P 1204140:1205456(1316) ack 1 win 4096
> 000938 IP src.rfe > bsd.12340: P 1205456:1206772(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1206772 win 65535
> 000640 IP src.rfe > bsd.12340: P 1206772:1208088(1316) ack 1 win 4096
> 000781 IP src.rfe > bsd.12340: P 1208088:1209404(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1209404 win 62903
> 001110 IP src.rfe > bsd.12340: P 1209404:1210720(1316) ack 1 win 4096
> 000780 IP src.rfe > bsd.12340: P 1210720:1212036(1316) ack 1 win 4096
> 000011 IP bsd.12340 > src.rfe: . ack 1212036 win 60271
> 000641 IP src.rfe > bsd.12340: P 1212036:1213352(1316) ack 1 win 4096
> 000782 IP src.rfe > bsd.12340: P 1213352:1214668(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1214668 win 57639
> 000953 IP src.rfe > bsd.12340: P 1214668:1215984(1316) ack 1 win 4096
> 000941 IP src.rfe > bsd.12340: P 1215984:1217300(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1217300 win 55007
> 000952 IP src.rfe > bsd.12340: P 1217300:1218616(1316) ack 1 win 4096
> 000781 IP src.rfe > bsd.12340: P 1218616:1219932(1316) ack 1 win 4096
> 000011 IP bsd.12340 > src.rfe: . ack 1219932 win 52375
> 000798 IP src.rfe > bsd.12340: P 1219932:1221248(1316) ack 1 win 4096
> 000794 IP src.rfe > bsd.12340: P 1221248:1222564(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1222564 win 49743
> 000646 IP src.rfe > bsd.12340: P 1222564:1223880(1316) ack 1 win 4096
> 000933 IP src.rfe > bsd.12340: P 1223880:1225196(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1225196 win 47111
> 000954 IP src.rfe > bsd.12340: P 1225196:1226512(1316) ack 1 win 4096
> 000625 IP src.rfe > bsd.12340: P 1226512:1227828(1316) ack 1 win 4096
> 000011 IP bsd.12340 > src.rfe: . ack 1227828 win 44479
> 000798 IP src.rfe > bsd.12340: P 1227828:1229144(1316) ack 1 win 4096
> 000936 IP src.rfe > bsd.12340: P 1229144:1230460(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1230460 win 41847
> 000953 IP src.rfe > bsd.12340: P 1230460:1231776(1316) ack 1 win 4096
> 000624 IP src.rfe > bsd.12340: P 1231776:1233092(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1233092 win 39215
> 000797 IP src.rfe > bsd.12340: P 1233092:1234408(1316) ack 1 win 4096
> 000780 IP src.rfe > bsd.12340: P 1234408:1235724(1316) ack 1 win 4096
> 000011 IP bsd.12340 > src.rfe: . ack 1235724 win 36583
> 000953 IP src.rfe > bsd.12340: P 1235724:1237040(1316) ack 1 win 4096
> 000937 IP src.rfe > bsd.12340: P 1237040:1238356(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1238356 win 33951
> 000641 IP src.rfe > bsd.12340: P 1238356:1239672(1316) ack 1 win 4096
> 001092 IP src.rfe > bsd.12340: P 1239672:1240988(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1240988 win 31319
> 000796 IP src.rfe > bsd.12340: P 1240988:1242304(1316) ack 1 win 4096
> 000626 IP src.rfe > bsd.12340: P 1242304:1243620(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1243620 win 28687
> 000952 IP src.rfe > bsd.12340: P 1243620:1244936(1316) ack 1 win 4096
> 000783 IP src.rfe > bsd.12340: P 1244936:1246252(1316) ack 1 win 4096
> 000015 IP bsd.12340 > src.rfe: . ack 1246252 win 26055
> 000797 IP src.rfe > bsd.12340: P 1246252:1247568(1316) ack 1 win 4096
> 000937 IP src.rfe > bsd.12340: P 1247568:1248884(1316) ack 1 win 4096
> 000011 IP bsd.12340 > src.rfe: . ack 1248884 win 23423
> 000797 IP src.rfe > bsd.12340: P 1248884:1250200(1316) ack 1 win 4096
> 000938 IP src.rfe > bsd.12340: P 1250200:1251516(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1251516 win 20791
> 000642 IP src.rfe > bsd.12340: P 1251516:1252832(1316) ack 1 win 4096
> 000782 IP src.rfe > bsd.12340: P 1252832:1254148(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1254148 win 18159
> 000797 IP src.rfe > bsd.12340: P 1254148:1255464(1316) ack 1 win 4096
> 001251 IP src.rfe > bsd.12340: P 1255464:1256780(1316) ack 1 win 4096
> 000014 IP bsd.12340 > src.rfe: . ack 1256780 win 15527
> 000640 IP src.rfe > bsd.12340: P 1256780:1258096(1316) ack 1 win 4096
> 000937 IP src.rfe > bsd.12340: P 1258096:1259412(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1259412 win 12895
> 000797 IP src.rfe > bsd.12340: P 1259412:1260728(1316) ack 1 win 4096
> 000945 IP src.rfe > bsd.12340: P 1260728:1262044(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1262044 win 10263
> 000640 IP src.rfe > bsd.12340: P 1262044:1263360(1316) ack 1 win 4096
> 000952 IP src.rfe > bsd.12340: P 1263360:1264676(1316) ack 1 win 4096
> 000014 IP bsd.12340 > src.rfe: . ack 1264676 win 7631
> 000796 IP src.rfe > bsd.12340: P 1264676:1265992(1316) ack 1 win 4096
> 000938 IP src.rfe > bsd.12340: P 1265992:1267308(1316) ack 1 win 4096
> 000011 IP bsd.12340 > src.rfe: . ack 1267308 win 4999
> 000801 IP src.rfe > bsd.12340: P 1267308:1268624(1316) ack 1 win 4096
> 000936 IP src.rfe > bsd.12340: P 1268624:1269940(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1269940 win 2367
> 000798 IP src.rfe > bsd.12340: P 1269940:1271256(1316) ack 1 win 4096
> 099460 IP bsd.12340 > src.rfe: . ack 1271256 win 1051			<-- window smaller than 1316
> 5. 537213 IP bsd.12340 > src.rfe: . ack 1271256 win 4096		<-- does this mean 5.5 seconds?  :-(
> 000027 IP bsd.12340 > src.rfe: . ack 1271256 win 31744
> 000026 IP bsd.12340 > src.rfe: . ack 1271256 win 57088
> 000024 IP bsd.12340 > src.rfe: . ack 1271256 win 65535
> 000333 IP src.rfe > bsd.12340: P 1271256:1272572(1316) ack 1 win 4096
> 000006 IP src.rfe > bsd.12340: P 1272572:1273888(1316) ack 1 win 4096
> 000034 IP bsd.12340 > src.rfe: . ack 1273888 win 65535
> 000113 IP src.rfe > bsd.12340: P 1273888:1275204(1316) ack 1 win 4096
> 000157 IP src.rfe > bsd.12340: P 1275204:1276520(1316) ack 1 win 4096
> 000003 IP src.rfe > bsd.12340: P 1271256:1272572(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1276520 win 65535
> 000008 IP bsd.12340 > src.rfe: . ack 1276520 win 65535
> 000134 IP src.rfe > bsd.12340: P 1272572:1273888(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1276520 win 65535
> 000144 IP src.rfe > bsd.12340: P 1273888:1275204(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1276520 win 65535
> 000145 IP src.rfe > bsd.12340: P 1275204:1276520(1316) ack 1 win 4096
> 000003 IP src.rfe > bsd.12340: P 1276520:1277836(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1276520 win 65535
> 000142 IP src.rfe > bsd.12340: P 1277836:1279152(1316) ack 1 win 4096
> 000013 IP bsd.12340 > src.rfe: . ack 1279152 win 65535
> 000152 IP src.rfe > bsd.12340: P 1279152:1280468(1316) ack 1 win 4096
> 000149 IP src.rfe > bsd.12340: P 1280468:1281784(1316) ack 1 win 4096
> 000003 IP src.rfe > bsd.12340: P 1276520:1277836(1316) ack 1 win 4096	<--- dup data?
> 000012 IP bsd.12340 > src.rfe: . ack 1281784 win 65535
> 000007 IP bsd.12340 > src.rfe: . ack 1281784 win 65535			<--- a dup ack after only 7 usec?
> 000135 IP src.rfe > bsd.12340: P 1277836:1279152(1316) ack 1 win 4096   <--- dup data even though it was acked?
> 000013 IP bsd.12340 > src.rfe: . ack 1281784 win 65535			<--- another dup ack?	
> 000144 IP src.rfe > bsd.12340: P 1279152:1280468(1316) ack 1 win 4096	<--- another dup data?
> 000003 IP src.rfe > bsd.12340: P 1280468:1281784(1316) ack 1 win 4096	<--- another dup data?
> 000012 IP bsd.12340 > src.rfe: . ack 1281784 win 65535			<--- and another dup ack?
> 000005 IP bsd.12340 > src.rfe: . ack 1281784 win 65535			<--- yet another dup ack?
> 000137 IP src.rfe > bsd.12340: P 1281784:1283100(1316) ack 1 win 4096
> 000158 IP src.rfe > bsd.12340: P 1283100:1284416(1316) ack 1 win 4096
> 000012 IP bsd.12340 > src.rfe: . ack 1284416 win 65535
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"
> 
> 
> 
> 
> 
> 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061118221846.3ae8c1ee.wmoran>