Date: Sun, 3 Jun 2012 05:18:14 +0000 From: Kevin Oberman <kob6558@gmail.com> To: Lawrence Stewart <lstewart@freebsd.org> Cc: freebsd-net@freebsd.org, Andrew Gallatin <gallatin@cs.duke.edu>, Andrew Gallatin <gallatin@myri.com> Subject: Re: Major performance hit with ToS setting Message-ID: <CAN6yY1v08qk2VhXFg0Qiz-pMM6md2c_E_kEvA-oqbxuvSN1JDg@mail.gmail.com> In-Reply-To: <4FC82D6C.4050309@freebsd.org> References: <CAN6yY1sLxFJ18ANO7nQqLetnJiT-K6pHC-X3yT1dWuWGa0VLUg@mail.gmail.com> <4FBF88CE.20209@cs.duke.edu> <CAN6yY1v%2Bvf=SW7WDGHxCkJtOdj8K3f450jNxFWK_Jc%2B-pFg0nA@mail.gmail.com> <4FC82D6C.4050309@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Jun 1, 2012 at 2:48 AM, Lawrence Stewart <lstewart@freebsd.org> wrote:
> On 05/31/12 13:33, Kevin Oberman wrote:
> [snip]
>>
>> I used SIFTR at the suggestion of Lawrence Stewart who headed the
>>
>> project to bring plugable congestion algorithms to FreeBSD and found
>> really odd congestion behavior. First, I do see a triple ACK, but the
>> congestion window suddenly drops from 73K to 8K. If I understand
>> CUBIC, it should half the congestion window, not what is happening..
>> It then increases slowly (in slow start) to 82K. while the slow-start
>> bytes are INCREASING, the congestion window again goes to 8K while the
>> SS size moves from 36K up to 52K. It just continues to bound wildly
>> between 8K (always the low point) and between 64k and 82K. The swings
>> start at 83K and, over the first few seconds the peaks drop to about
>> 64K.
>
>
> Oh, and a comment about this behaviour. Dropping back to 8k (1MSS) is only
> nasty if the TF_{CONG|FAST}RECOVERY flags are *not* set i.e. if you see cwnd
> grow, drop to 8k with those flags set, and then when the flags are unset,
> cwnd starts at the value of ssthresh, then that is perfectly normal recovery
> behaviour. What *is* nasty is if an RTO fires, which will reset cwnd to 8k,
> ssthresh to 2*MSS and make the connection effectively start from scratch
> again.
>
> There is evidence of RTOs in your siftr output, which is bad news e.g here's
> one example of 2 side-by-side log lines from your trace:
>
> # Direction,time,ssthresh,cwnd,flags
> i,1338319593.574706,27044,27044,1630544864
> o,1338319593.831482,16384,8192,1092625377
>
> Note the 300ms gap, and how cwnd resets to 1MSS and flags go from 1630544864
> (TF_WASCRECOVERY|TF_CONGRECOVERY|TF_WASFRECOVERY|TF_FASTRECOVERY) to
> 1092625377 (TF_WASCRECOVERY|TF_WASFRECOVERY).
What can I say but that you are right. When I looked at the interface
stats I found that the link overflow drops were through the roof! This
confuses me a bit since the traffic is outbound and I woudl assume
from the description on hte Myricom web page that these are input
drops. A problem a problem with that card? On systems that are
working "normally", I still see a sharp drop with the ToS bits set,
but nothing nearly as drastic. Now it is a drop from 4.5G to 728M on a
cross-country (US) circuit.
I am now looking for issues on the route that might explain the
performance, but the question of why the drop-of only shows up in
FreeBSD 8 means something odd is still going on. It is even possible
that the problem is with 7 and the losses are due to the policy for
ToS 32 on the path. ToS 32 is less than best effort in our network.
Maybe the marking was getting lost on 7. Not likely, but possible.
I'll post further details after some very careful reviews of router statistics.
Thanks, Lawrence!
--
R. Kevin Oberman, Network Engineer
E-mail: kob6558@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1v08qk2VhXFg0Qiz-pMM6md2c_E_kEvA-oqbxuvSN1JDg>