Date: Wed, 17 Jul 2019 20:00:58 +0200 From: Michael Tuexen <tuexen@freebsd.org> To: Kevin Bowling <kevin.bowling@kev009.com> Cc: Paul <devgs@ukr.net>, Vitalij Satanivskij <satan@ukr.net>, freebsd-net@freebsd.org Subject: Re: Issues with TCP Timestamps allocation Message-ID: <B20B549B-F169-4C97-A8B5-2B95B66EBA34@freebsd.org> In-Reply-To: <CAK7dMtC1FOD=_4%2B7ZE3B8gmuw3TOk5cCvOjdfPoBx8gD69VgsA@mail.gmail.com> References: <1562599181.734953000.1l9a1d23@frv39.fwdcdn.com> <0C475A01-9BCD-4E4A-9731-09AB919CA9BE@freebsd.org> <1562676414.933145000.z3zteyqp@frv39.fwdcdn.com> <1E9F3F99-C3E9-44DD-AA70-9B11E19D4769@freebsd.org> <20190717074243.GA65665@hell.ukr.net> <CE7F8390-6F56-4550-A8B3-84E0AC31C27C@freebsd.org> <20190717100926.GA24984@hell.ukr.net> <48817BF6-AEDD-4D28-95F8-A4D53E4999B1@freebsd.org> <20190717115502.GA53155@hell.ukr.net> <8763FDC7-8B71-41C3-8D1C-10416DA9A871@freebsd.org> <20190717123251.GA53638@hell.ukr.net> <F1CF6E29-E2F5-44FE-B904-DCE93FA91817@freebsd.org> <CAK7dMtC1FOD=_4%2B7ZE3B8gmuw3TOk5cCvOjdfPoBx8gD69VgsA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 17. Jul 2019, at 18:09, Kevin Bowling <kevin.bowling@kev009.com> = wrote: >=20 > Any knowledge of the endpoints, Linux boxes misconfigured with = tcp_tw_recycle? I contacted some Linux guys and they told me that tcp_tw_recycle is = specific to the 5 tuple. Is that not correct? The servers can be running Linux, I haven't checked all of them... For the problem we are seeing here, it port numbers are irrelevant. If = the server sends a FIN (and therefore goes to TIMEWAIT), one can experience = the problem, even if the client changes the port number and even if you talk to other server ports. I tested with the ssh port in addition to the web traffic. Best regards Michael >=20 > On Wed, Jul 17, 2019 at 5:42 AM Michael Tuexen <tuexen@freebsd.org> = wrote: > > On 17. Jul 2019, at 14:32, Vitalij Satanivskij <satan@ukr.net> = wrote: > >=20 > > Hmm, looks like with some host's work but not with another > >=20 > > Wed/17.07:/home/satan > > hell:-1522/15:28>curl https://volia.com > /dev/null > > % Total % Received % Xferd Average Speed Time Time = Time Current > > Dload Upload Total Spent = Left Speed > > 100 41519 0 41519 0 0 137k 0 --:--:-- --:--:-- = --:--:-- 137k > > Wed/17.07:/home/satan > > hell:-1523/15:28>curl https://volia.com > /dev/null > > % Total % Received % Xferd Average Speed Time Time = Time Current > > Dload Upload Total Spent = Left Speed > > 0 0 0 0 0 0 0 0 --:--:-- 0:00:53 = --:--:-- 0^C > > Wed/17.07:/home/satan > > hell:-1524/15:29>sysctl net.inet.tcp.rexmit_drop_options > > net.inet.tcp.rexmit_drop_options: 1 > OK, I can confirm that for https://volia.com only a timeout helps. >=20 > What I observed for now is that for the "blocking" to occur is it = crucial that > the server sends the FIN and therefore goes into the TIMEWAIT state. = The timeout > seems to be 60 seconds. > The blocking is also not limited to a single server port. >=20 > I'm not sure yet whether it is a broken end point or a broken middle = box. >=20 > Best regards > Michael > >=20 > > But=20 > >=20 > > MT> Interesting. It works for me: > > MT>=20 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 33637 0 --:--:-- --:--:-- = --:--:-- 33575 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 4834 0 --:--:-- 0:00:03 = --:--:-- 4833 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 35813 0 --:--:-- --:--:-- = --:--:-- 35813 > > MT> tuexen@head:~ % time curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 48320 0 --:--:-- --:--:-- = --:--:-- 48320 > > MT> 0.012u 0.031s 0:00.39 10.2% 140+245k 0+0io 0pf+0w > > MT> tuexen@head:~ % time curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 4592 0 --:--:-- 0:00:03 = --:--:-- 4591 > > MT> 0.031u 0.010s 0:03.99 1.0% 80+140k 0+0io 0pf+0w > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 37815 0 --:--:-- --:--:-- = --:--:-- 37737 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 27261 0 --:--:-- --:--:-- = --:--:-- 27220 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 4533 0 --:--:-- 0:00:04 = --:--:-- 4533 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 48320 0 --:--:-- --:--:-- = --:--:-- 48192 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 4746 0 --:--:-- 0:00:03 = --:--:-- 4745 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 4500 0 --:--:-- 0:00:04 = --:--:-- 4767 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 4726 0 --:--:-- 0:00:03 = --:--:-- 4726 > > MT> tuexen@head:~ % curl https://vitagramma.com > /dev/null > > MT> % Total % Received % Xferd Average Speed Time Time = Time Current > > MT> Dload Upload Total Spent = Left Speed > > MT> 100 18265 0 18265 0 0 34268 0 --:--:-- --:--:-- = --:--:-- 34332 > > MT> tuexen@head:~ %=20 > > MT>=20 > > MT> So it either works immediately or with a delay of 3 to 4 = seconds... > > MT>=20 > > MT> Best regards > > MT> Michael > > MT> >=20 > > MT> >=20 > > MT> > MT>=20 > > MT> > MT> Option 2: Disable the TCP timestamps (and window scaling) > > MT> > MT> To enable this, you configure on the client > > MT> > MT> sudo sysctl -w net.inet.tcp.rfc1323=3D0 > > MT> > MT> or put > > MT> > MT> net.inet.tcp.rfc1323=3D0 > > MT> > MT> in /etc/sysctl.conf > > MT> > MT> and reboot. > > MT> > MT> This disables the timestamp option and window scaling = completely. This allows you to > > MT> > MT> setup the connections without any delay. However, you = don't have the benefits of the > > MT> > MT> extension. > > MT> > MT>=20 > > MT> > MT> Both options don't require any code changes. > > MT> >=20 > > MT> > This option was tested some time before. Yep it's help. But = overal performance of tcp networking ... Let's say to bad :( > > MT> >=20 > > MT> >=20 > > MT> >=20 > > MT> >=20 > > MT> > MT> Best regards > > MT> > MT> Michael > > MT> > MT>=20 > > MT> > MT>=20 > > MT> > MT> >=20 > > MT> > MT> >=20 > > MT> > MT> > MT>=20 > > MT> > MT> > MT> Best regards > > MT> > MT> > MT> Michael > > MT> > MT> > MT> >=20 > > MT> > MT> > MT> >=20 > > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > Michael Tuexen wrote: > > MT> > MT> > MT> > MT>=20 > > MT> > MT> > MT> > MT>=20 > > MT> > MT> > MT> > MT> > On 9. Jul 2019, at 14:58, Paul = <devgs@ukr.net> wrote: > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> > Hi Michael, > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> > 9 July 2019, 15:34:29, by "Michael Tuexen" = <tuexen@freebsd.org>: > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >>> On 8. Jul 2019, at 17:22, Paul = <devgs@ukr.net> wrote: > > MT> > MT> > MT> > MT> >>>=20 > > MT> > MT> > MT> > MT> >>>=20 > > MT> > MT> > MT> > MT> >>>=20 > > MT> > MT> > MT> > MT> >>> 8 July 2019, 17:12:21, by "Michael Tuexen" = <tuexen@freebsd.org>: > > MT> > MT> > MT> > MT> >>>=20 > > MT> > MT> > MT> > MT> >>>>> On 8. Jul 2019, at 15:24, Paul = <devgs@ukr.net> wrote: > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>> Hi Michael, > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>> 8 July 2019, 15:53:15, by "Michael = Tuexen" <tuexen@freebsd.org>: > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> On 8. Jul 2019, at 12:37, Paul = <devgs@ukr.net> wrote: > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> Hi team, > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> Recently we had an upgrade to 12 = Stable. Immediately after, we have started=20 > > MT> > MT> > MT> > MT> >>>>>>> seeing some strange connection = establishment timeouts to some fixed number > > MT> > MT> > MT> > MT> >>>>>>> of external (world) hosts. The issue = was persistent and easy to reproduce. > > MT> > MT> > MT> > MT> >>>>>>> Thanks to a patience and dedication of = our system engineer we have tracked =20 > > MT> > MT> > MT> > MT> >>>>>>> this issue down to a specific commit: > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> = https://svnweb.freebsd.org/base?view=3Drevision&revision=3D338053 > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> This patch was also back-ported into = 11 Stable: > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> = https://svnweb.freebsd.org/base?view=3Drevision&revision=3D348435 > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> Among other things this patch changes = the timestamp allocation strategy, > > MT> > MT> > MT> > MT> >>>>>>> by introducing a deterministic = randomness via a hash function that takes > > MT> > MT> > MT> > MT> >>>>>>> into account a random key as well as = source address, source port, dest > > MT> > MT> > MT> > MT> >>>>>>> address and dest port. As the result, = timestamp offsets of different > > MT> > MT> > MT> > MT> >>>>>>> tuples (SA,SP,DA,DP) will be wildly = different and will jump from small=20 > > MT> > MT> > MT> > MT> >>>>>>> to large numbers and back, as long as = something in the tuple changes. > > MT> > MT> > MT> > MT> >>>>>> Hi Paul, > > MT> > MT> > MT> > MT> >>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>> this is correct. > > MT> > MT> > MT> > MT> >>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>> Please note that the same happens with = the old method, if two hosts with > > MT> > MT> > MT> > MT> >>>>>> different uptimes are bind a consumer = grade NAT. > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>> If NAT does not replace timestamps then = yes, it should be the case. > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> After performing various tests of = hosts that produce the above mentioned=20 > > MT> > MT> > MT> > MT> >>>>>>> issue we came to conclusion that there = are some interesting implementations=20 > > MT> > MT> > MT> > MT> >>>>>>> that drop SYN packets with timestamps = smaller than the largest timestamp=20 > > MT> > MT> > MT> > MT> >>>>>>> value from streams of all recent or = current connections from a specific=20 > > MT> > MT> > MT> > MT> >>>>>>> address. This looks as some kind of = SYN flood protection. > > MT> > MT> > MT> > MT> >>>>>> This also breaks multiple hosts with = different uptimes behind a consumer > > MT> > MT> > MT> > MT> >>>>>> level NAT talking to such a server. > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> To ensure that each external host is = not going to see a wild jumps of=20 > > MT> > MT> > MT> > MT> >>>>>>> timestamp values I propose a patch = that removes ports from the equation > > MT> > MT> > MT> > MT> >>>>>>> all together, when calculating the = timestamp offset: > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> Index: sys/netinet/tcp_subr.c > > MT> > MT> > MT> > MT> >>>>>>> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > MT> > MT> > MT> > MT> >>>>>>> --- sys/netinet/tcp_subr.c = (revision 348435) > > MT> > MT> > MT> > MT> >>>>>>> +++ sys/netinet/tcp_subr.c = (working copy) > > MT> > MT> > MT> > MT> >>>>>>> @@ -2224,7 +2224,22 @@ > > MT> > MT> > MT> > MT> >>>>>>> uint32_t > > MT> > MT> > MT> > MT> >>>>>>> tcp_new_ts_offset(struct in_conninfo = *inc) > > MT> > MT> > MT> > MT> >>>>>>> { > > MT> > MT> > MT> > MT> >>>>>>> - return (tcp_keyed_hash(inc, = V_ts_offset_secret)); > > MT> > MT> > MT> > MT> >>>>>>> + /*=20 > > MT> > MT> > MT> > MT> >>>>>>> + * Some implementations show = a strange behaviour when a wildly random=20 > > MT> > MT> > MT> > MT> >>>>>>> + * timestamps allocated for = different streams. It seems that only the > > MT> > MT> > MT> > MT> >>>>>>> + * SYN packets are affected. = Observed implementations drop SYN packets > > MT> > MT> > MT> > MT> >>>>>>> + * with timestamps smaller = than the largest timestamp value of all=20 > > MT> > MT> > MT> > MT> >>>>>>> + * recent or current = connections from specific a address. To mitigate=20 > > MT> > MT> > MT> > MT> >>>>>>> + * this we are going to = ensure that each host will always observe=20 > > MT> > MT> > MT> > MT> >>>>>>> + * timestamps as increasing = no matter the stream: by dropping ports > > MT> > MT> > MT> > MT> >>>>>>> + * from the equation. > > MT> > MT> > MT> > MT> >>>>>>> + */=20 > > MT> > MT> > MT> > MT> >>>>>>> + struct in_conninfo inc_copy =3D= *inc; > > MT> > MT> > MT> > MT> >>>>>>> + > > MT> > MT> > MT> > MT> >>>>>>> + inc_copy.inc_fport =3D 0; > > MT> > MT> > MT> > MT> >>>>>>> + inc_copy.inc_lport =3D 0; > > MT> > MT> > MT> > MT> >>>>>>> + > > MT> > MT> > MT> > MT> >>>>>>> + return = (tcp_keyed_hash(&inc_copy, V_ts_offset_secret)); > > MT> > MT> > MT> > MT> >>>>>>> } > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> /* > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> In any case, the solution of the = uptime leak, implemented in rev338053 is=20 > > MT> > MT> > MT> > MT> >>>>>>> not going to suffer, because a = supposed attacker is currently able to use=20 > > MT> > MT> > MT> > MT> >>>>>>> any fixed values of SP and DP, albeit = not 0, anyway, to remove them out=20 > > MT> > MT> > MT> > MT> >>>>>>> of the equation. > > MT> > MT> > MT> > MT> >>>>>> Can you describe how a peer can compute = the uptime from two observed timestamps? > > MT> > MT> > MT> > MT> >>>>>> I don't see how you can do that... > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>> Supposed attacker could run a script = that continuously monitors timestamps, > > MT> > MT> > MT> > MT> >>>>> for example via a periodic TCP = connection from a fixed local port (eg 12345)=20 > > MT> > MT> > MT> > MT> >>>>> and a fixed local address to the fixed = victim's address and port (eg 80). > > MT> > MT> > MT> > MT> >>>>> Whenever large discrepancy is observed, = attacker can assume that reboot has=20 > > MT> > MT> > MT> > MT> >>>>> happened (due to V_ts_offset_secret = re-generation), hence the received=20 > > MT> > MT> > MT> > MT> >>>>> timestamp is considered an approximate = point of reboot from which the uptime > > MT> > MT> > MT> > MT> >>>>> can be calculated, until the next reboot = and so on. > > MT> > MT> > MT> > MT> >>>> Ahh, I see. The patch we are talking = about is not intended to protect against > > MT> > MT> > MT> > MT> >>>> continuous monitoring, which is something = you can always do. You could even > > MT> > MT> > MT> > MT> >>>> watch for service availability and detect = reboots. A change of the local key > > MT> > MT> > MT> > MT> >>>> would also look similar to a reboot = without a temporary loss of connectivity. > > MT> > MT> > MT> > MT> >>>>=20 > > MT> > MT> > MT> > MT> >>>> Thanks for the clarification. > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> There is the list of example hosts = that we were able to reproduce the=20 > > MT> > MT> > MT> > MT> >>>>>>> issue with: > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> curl -v http://88.99.60.171:80 > > MT> > MT> > MT> > MT> >>>>>>> curl -v http://163.172.71.252:80 > > MT> > MT> > MT> > MT> >>>>>>> curl -v http://5.9.242.150:80 > > MT> > MT> > MT> > MT> >>>>>>> curl -v https://185.134.205.105:443 > > MT> > MT> > MT> > MT> >>>>>>> curl -v https://136.243.1.231:443 > > MT> > MT> > MT> > MT> >>>>>>> curl -v https://144.76.196.4:443 > > MT> > MT> > MT> > MT> >>>>>>> curl -v http://94.127.191.194:80 > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> To reproduce, call curl repeatedly = with a same URL some number of times.=20 > > MT> > MT> > MT> > MT> >>>>>>> You are going to see some of the = requests stuck in=20 > > MT> > MT> > MT> > MT> >>>>>>> `* Trying XXX.XXX.XXX.XXX...` > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> For some reason, the easiest way to = reproduce the issue is with nc: > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> $ echo "foooooo" | nc -v 88.99.60.171 = 80 > > MT> > MT> > MT> > MT> >>>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>> Only a few such calls are required = until one of them is stuck on connect(): > > MT> > MT> > MT> > MT> >>>>>>> issuing SYN packets with an = exponential backoff. > > MT> > MT> > MT> > MT> >>>>>> Thanks for providing an end-point to = test with. I'll take a look. > > MT> > MT> > MT> > MT> >>>>>> Just to be clear: You are running a = FreeBSD client against one of the above > > MT> > MT> > MT> > MT> >>>>>> servers and experience the problem with = the new timestamp computations. > > MT> > MT> > MT> > MT> >>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>> You are not running arbitrary clients = against a FreeBSD server... > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>> We are talking about FreeBSD being the = client. Peers that yield this unwanted > > MT> > MT> > MT> > MT> >>>>> behaviour are unknown. Little bit of = tinkering showed that some of them run=20 > > MT> > MT> > MT> > MT> >>>>> Debian: > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>> telnet 88.99.60.171 22 > > MT> > MT> > MT> > MT> >>>>> Trying 88.99.60.171... > > MT> > MT> > MT> > MT> >>>>> Connected to 88.99.60.171. > > MT> > MT> > MT> > MT> >>>>> Escape character is '^]'. > > MT> > MT> > MT> > MT> >>>>> SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3 > > MT> > MT> > MT> > MT> >>>> Also some are hosted by Hetzner, but not = all. I'll will look into > > MT> > MT> > MT> > MT> >>>> this tomorrow, since I'm on a deadline = today (well it is 2am tomorrow > > MT> > MT> > MT> > MT> >>>> morning, to be precise)... > > MT> > MT> > MT> > MT> >>>=20 > > MT> > MT> > MT> > MT> >>> Thanks a lot, I would appreciate that. > > MT> > MT> > MT> > MT> >> Hi Paul, > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> I have looked into this. > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> * The FreeBSD behaviour is the one which is = specified in the last bullet item > > MT> > MT> > MT> > MT> >> in = https://tools.ietf.org/html/rfc7323#section-5.4 > > MT> > MT> > MT> > MT> >> It is also the one, which is RECOMMENDED = in > > MT> > MT> > MT> > MT> >> = https://tools.ietf.org/html/rfc7323#section-7.1=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> * My NAT box (a popular one in Germany) = does NOT rewrite TCP timestamps. > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> This means that the host you are referring = to have some sort of protection, > > MT> > MT> > MT> > MT> >> which makes incorrect assumptions. It will = also break multiple hosts behind > > MT> > MT> > MT> > MT> >> a NAT. > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> I can run > > MT> > MT> > MT> > MT> >> curl -v http://88.99.60.171:80 > > MT> > MT> > MT> > MT> >> in a loop without any problems from a = FreeBSD head system. I tested 1000 > > MT> > MT> > MT> > MT> >> iterations or so. The TS.val is jumping up = and down as expected. > > MT> > MT> > MT> > MT> >> I'm wondering why you are observing errors = in this case, too. > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> However, doing something like > > MT> > MT> > MT> > MT> >> echo "foooooo" | nc -v 88.99.60.171 80 > > MT> > MT> > MT> > MT> >> triggers the problem. > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> So I think there is some functionality (in = a middlebox or running on the host), > > MT> > MT> > MT> > MT> >> which incorrectly assume monotonic = timestamps between multiple TCP connections > > MT> > MT> > MT> > MT> >> coming from the same IP address, but only = in case of errors at the application layer. > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> > Yeah, exactly, some hosts seem to enable = this only in case of an error in HTTP > > MT> > MT> > MT> > MT> > communication (some smart proxy?). However, = there are some that behave this way > > MT> > MT> > MT> > MT> > regardless of errors, for example these: > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> > curl -v https://185.134.205.105:443 > > MT> > MT> > MT> > MT> > curl -v https://136.243.1.231:443 > > MT> > MT> > MT> > MT> Wireshark sees an Encrypted Alert in both = cases. So I guess this is another indication > > MT> > MT> > MT> > MT> of "error at the application layer". > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> Do you have any insights whether the hosts = you are listed share something in > > MT> > MT> > MT> > MT> >> common. Some of them are hosted by Hetzner, = but not all. > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> > Nope. A whole set of endpoints that we have = detected so far is pretty diverse, > > MT> > MT> > MT> > MT> > containing a lot of different locations = geographically, as well as different > > MT> > MT> > MT> > MT> > hosters. > > MT> > MT> > MT> > MT> OK. Thanks for the clarification. > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> I think in general, it is the correct thing = to include the port numbers in > > MT> > MT> > MT> > MT> >> the offset computation. We might add a = sysctl variable to control the inclusion. > > MT> > MT> > MT> > MT> >> This would allow interworking with broken = middleboxes. > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> > Yeah, I completely agree that these rare = cases should not dictate the implementation. > > MT> > MT> > MT> > MT> > But an ability to enable a work-around via = sysctl would be greatly appreciated. > > MT> > MT> > MT> > MT> > Currently we are unable to roll-out the = upgrade across all servers because of this > > MT> > MT> > MT> > MT> > issue: even though it happens not so often, = a lot of requests from our users=20 > > MT> > MT> > MT> > MT> > get stuck or fail all together. For example, = a host 185.134.205.105 is a kind of > > MT> > MT> > MT> > MT> > social network that our proxy servers = connect to so securely access to content, > > MT> > MT> > MT> > MT> > such as images, on behalf of our users. > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> Please note, this does not fix the case of = multiple clients behind a NAT. > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> > Yeah, that's true. Fortunately we don't use = NAT. > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> I'm also trying to figure out how and why = Linux and Windows are handling this. > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> > Thanks for bothering! > > MT> > MT> > MT> > MT> Will let you know what I figure out. > > MT> > MT> > MT> > MT>=20 > > MT> > MT> > MT> > MT> Best regards > > MT> > MT> > MT> > MT> Michael > > MT> > MT> > MT> > MT> >=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >> Best regards > > MT> > MT> > MT> > MT> >> Michael > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >>>=20 > > MT> > MT> > MT> > MT> >>>>=20 > > MT> > MT> > MT> > MT> >>>> Best regards > > MT> > MT> > MT> > MT> >>>> Michael=20 > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>> Best regards > > MT> > MT> > MT> > MT> >>>>>> Michael > > MT> > MT> > MT> > MT> >>>>>>=20 > > MT> > MT> > MT> > MT> >>>>>>=20 > > MT> > MT> > MT> > MT> >>>>=20 > > MT> > MT> > MT> > MT> >>>>=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT> >>=20 > > MT> > MT> > MT> > MT>=20 > > MT> > MT> > MT> > MT> = _______________________________________________ > > MT> > MT> > MT> > MT> freebsd-net@freebsd.org mailing list > > MT> > MT> > MT> > MT> = https://lists.freebsd.org/mailman/listinfo/freebsd-net > > MT> > MT> > MT> > MT> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" > > MT> > MT> > MT>=20 > > MT> > MT> > MT> _______________________________________________ > > MT> > MT> > MT> freebsd-net@freebsd.org mailing list > > MT> > MT> > MT> = https://lists.freebsd.org/mailman/listinfo/freebsd-net > > MT> > MT> > MT> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" > > MT> > MT>=20 > > MT> > _______________________________________________ > > MT> > freebsd-net@freebsd.org mailing list > > MT> > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > MT> > To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" > > MT>=20 > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B20B549B-F169-4C97-A8B5-2B95B66EBA34>