Date: Sun, 19 Feb 2017 01:37:35 +0000 From: Steven Hartland <steven.hartland@multiplay.co.uk> To: Lawrence Stewart <lstewart@freebsd.org>, freebsd-transport@freebsd.org Subject: Re: TCP Receive buffer scaling without Timestamps Message-ID: <2cc38c5d-78a9-4ce1-902e-70f2e55f14a2@multiplay.co.uk> In-Reply-To: <be22c783-26d8-321d-d7b1-1edeb27d11e8@freebsd.org> References: <b9906198-a026-1403-73e6-279066b4e197@multiplay.co.uk> <be22c783-26d8-321d-d7b1-1edeb27d11e8@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 18/02/2017 00:33, Lawrence Stewart wrote:
> On 18/02/2017 09:37, Steven Hartland wrote:
> [snip]
>> So the questions:
>>
>> 1. Has anyone looked at this issue before or is working on it?
>> 2. Do people think we could add support for RTT estimates and enable
>> receive scaling without major tcp stack changes?
> We already have a non-TS-enabled RTT estimator in place c.f. the
> relevant code in tcp_input.c:
>
> if ((to.to_flags & TOF_TS) != 0 &&
> to.to_tsecr) {
> uint32_t t;
>
> t = tcp_ts_getticks() - to.to_tsecr;
> if (!tp->t_rttlow || tp->t_rttlow > t)
> tp->t_rttlow = t;
> tcp_xmit_timer(tp,
> TCP_TS_TO_TICKS(t) + 1);
> } else if (tp->t_rtttime &&
> SEQ_GT(th->th_ack, tp->t_rtseq)) {
> if (!tp->t_rttlow ||
> tp->t_rttlow > ticks - tp->t_rtttime)
> tp->t_rttlow = ticks - tp->t_rtttime;
> tcp_xmit_timer(tp,
> ticks - tp->t_rtttime);
> }
>
>
> The autoscaling implementation just needs to have its insistence on
> using timestamps stuffed into a cannon and shot in the direction of the
> sun. This is the relevant block of rcvbuf code from tcp_input.c:
>
>
> if (V_tcp_do_autorcvbuf &&
> (to.to_flags & TOF_TS) &&
> to.to_tsecr &&
> (so->so_rcv.sb_flags & SB_AUTOSIZE)) {
> if (TSTMP_GT(to.to_tsecr, tp->rfbuf_ts) &&
> to.to_tsecr - tp->rfbuf_ts < hz) {
> if (tp->rfbuf_cnt >
> (so->so_rcv.sb_hiwat / 8 * 7) &&
> so->so_rcv.sb_hiwat <
> V_tcp_autorcvbuf_max) {
> newsize =
> min(so->so_rcv.sb_hiwat +
> V_tcp_autorcvbuf_inc,
> V_tcp_autorcvbuf_max);
> }
> /* Start over with next RTT. */
> tp->rfbuf_ts = 0;
> tp->rfbuf_cnt = 0;
> } else
> tp->rfbuf_cnt += tlen; /* add up */
> }
>
> It's pretty trivial to fix by anyone so inclined.
Thanks for the pointers Lawrence, I'm not very familiar with network
stack but I've had a stab at this, after reading through the current
code in the blocks you pointed out, along side some experimentation ;-)
The results seem pretty encouraging, with S3 downloads with a ~17ms
latency on a 1Gbps jumping from ~3MB/s to ~80MB/s.
I'd appreciate feedback on the review which can be found here:
https://reviews.freebsd.org/D9668
Its a bit rough ATM in order to facilitate monitoring the behaviour with
dtrace.
My initial attempt did try to replicate the tests done to facilitate the
calls to tcp_xmit_timer, but they couldn't be reused due to the changes
the tcp_xmit_timer calls themselves trigger, so I based it off the
already calculated and stored smoothed RTT instead.
Regards
Steve
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2cc38c5d-78a9-4ce1-902e-70f2e55f14a2>