Date: Mon, 29 Dec 2014 22:03:38 +0100 From: Nikolay Denev <ndenev@gmail.com> To: Adrian Chadd <adrian@freebsd.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: setfib and RSTs Message-ID: <CA%2BP_MZGa8s9PzhoA=kY8YRcG5kRsVb=oP8pjtOXV%2BDH_meG2nQ@mail.gmail.com> In-Reply-To: <CAJ-Vmo=B3c9p6nMpHA6z75oKBErTrX0Ru8MDhHw1MgN=Ts_%2B_w@mail.gmail.com> References: <CA%2BP_MZHi1hMGq8qbOvpYSMLk4Usr4CAXjR1PdKRt7dE_vMJ7UQ@mail.gmail.com> <54A0FDD9.4090009@freebsd.org> <CA%2BP_MZGsX3_UB2wigrD-WWB0aN=bSf9Xxw147y2vWLAJ6_syng@mail.gmail.com> <CAJ-Vmo=B3c9p6nMpHA6z75oKBErTrX0Ru8MDhHw1MgN=Ts_%2B_w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Dec 29, 2014 at 7:47 PM, Adrian Chadd <adrian@freebsd.org> wrote: > Have you filed a PR about this? This seems like it's pretty important to > fix. > > (For other non-torrent reasons too; like people wishing to use this > for things like tor browsing.) > > > -adrian > > > On 29 December 2014 at 05:11, Nikolay Denev <ndenev@gmail.com> wrote: > > On Mon, Dec 29, 2014 at 8:08 AM, Julian Elischer <julian@freebsd.org> > wrote: > > > >> On 12/26/14 10:41 PM, Nikolay Denev wrote: > >> > >>> Hi, > >>> > >>> I have a process (bittorrent client) running in a non-default fib and > >>> using > >>> a VPN for default gateway: > >>> > >>> from /etc/rc.local : > >>> > >>> /usr/sbin/setfib 1 route add $vpn_provider 10.0.0.1 > >>> /usr/sbin/setfib 1 /usr/local/sbin/openvpn --config > >>> /usr/local/etc/openvpn/provider.ovpn > >>> /usr/sbin/setfib 1 /usr/sbin/service transmission onestart > >>> > >>> Then openvpn installs default gateway in fib 1 to point to the tun(4) > >>> interface. > >>> > >>> Stil, I'm seeing RST packets from the bittorrent client process to be > sent > >>> not via the tunnel, but to the default gateway of the lan which seems > >>> wrong. As if when the kernel generates the RST it does not take into > >>> account the FIB of the socket? > >>> > >> it's possible that you are correct. > >> I checked that RST and other generated packets used the FIB for the > >> session if it existed when they are generated, > >> but I don't know what they do when a single unexpected packet > enters..You > >> may need toset the fib in received packets using either the ipfw setfib > >> command or the ifconfig fib command. The RST should then use the same > FIB > >> to respond. > >> > >> let me know.. > >> > >> > >> Any ideas? > >>> _______________________________________________ > >>> freebsd-net@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net > >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >>> > >>> > >>> > >> > > > > The weird thing was that the RST was a response to a SYN-ACK. > > What I saw was the transmission bittorent daemon to try to connect to > some > > peer over tun2 (the openvpn connection), > > so it sends the SYN, the peer responds with SYN-ACK, then I see RST back > to > > the peer, but not over the tun2, but over lagg0 (as originally > mentioned). > > > > Here 10.8.0.31 is the tun2 address, and XX.XX.XX.XX is one of the peers > > that I was able to capture this happening. (both SYN-ACK and RST's are > > retransmitted, as the RSTs get blocked by the pf running on the default > gw > > as it does not have state for them, as the VPN is a running on a host > > inside the network) > > > > # Captured on tun2: > > > > [14:03][ndenev@macbook:~]%tcpdump -vnr 1.pcap host XX.XX.XX.XX > > reading from file 1.pcap, link-type NULL (BSD loopback) > > 20:43:39.623920 IP (tos 0x0, ttl 64, id 12450, offset 0, flags [DF], > proto > > TCP (6), length 60) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [S], cksum 0xace8 > (correct), > > seq 2985341180, win 8192, options [mss 1460,nop,wscale 6,sackOK,TS val > > 515982338 ecr 0], length 0 > > 20:43:39.912558 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x5a3d > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207923992 ecr 515982338,nop,wscale 7], length 0 > > 20:43:41.132751 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x590a > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207924299 ecr 515982338,nop,wscale 7], length 0 > > 20:43:43.141474 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x5716 > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207924799 ecr 515982338,nop,wscale 7], length 0 > > 20:43:47.129657 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x532e > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207925799 ecr 515982338,nop,wscale 7], length 0 > > 20:43:55.133735 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x4b5e > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207927799 ecr 515982338,nop,wscale 7], length 0 > > 20:44:11.134635 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x3bbe > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207931799 ecr 515982338,nop,wscale 7], length 0 > > 20:45:11.290740 IP (tos 0x0, ttl 64, id 61675, offset 0, flags [DF], > proto > > TCP (6), length 60) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [S], cksum 0xdbfb > (correct), > > seq 2733189104, win 8192, options [mss 1460,nop,wscale 6,sackOK,TS val > > 516074004 ecr 0], length 0 > > 20:45:11.595504 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.61382: Flags [S.], cksum 0x81f1 > > (correct), seq 2072870049, ack 2733189105, win 28960, options [mss > > 1368,sackOK,TS val 207946912 ecr 516074004,nop,wscale 7], length 0 > > 20:45:12.748574 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.61382: Flags [S.], cksum 0x80cf > > (correct), seq 2072870049, ack 2733189105, win 28960, options [mss > > 1368,sackOK,TS val 207947202 ecr 516074004,nop,wscale 7], length 0 > > 20:45:14.746692 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.61382: Flags [S.], cksum 0x7edb > > (correct), seq 2072870049, ack 2733189105, win 28960, options [mss > > 1368,sackOK,TS val 207947702 ecr 516074004,nop,wscale 7], length 0 > > > > # Captured on lagg0: > > > > [14:04][ndenev@macbook:~]%tcpdump -vnr 2.pcap host XX.XX.XX.XX > > reading from file 2.pcap, link-type EN10MB (Ethernet) > > 20:43:39.912570 IP (tos 0x0, ttl 64, id 12524, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->a2c)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:43:41.132761 IP (tos 0x0, ttl 64, id 13085, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->7fb)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:43:43.141485 IP (tos 0x0, ttl 64, id 14702, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->1aa)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:43:47.129669 IP (tos 0x0, ttl 64, id 16915, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->f904)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:43:55.133748 IP (tos 0x0, ttl 64, id 23534, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->df29)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:44:11.134645 IP (tos 0x0, ttl 64, id 30066, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->c5a5)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:45:11.595514 IP (tos 0x0, ttl 64, id 62112, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->4877)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:12.748583 IP (tos 0x0, ttl 64, id 62741, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->4602)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:14.746703 IP (tos 0x0, ttl 64, id 63874, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->4195)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:18.754657 IP (tos 0x0, ttl 64, id 65318, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->3bf1)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:26.759522 IP (tos 0x0, ttl 64, id 3473, offset 0, flags [DF], proto > > TCP (6), length 40, bad cksum 0 (->2d87)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:42.758524 IP (tos 0x0, ttl 64, id 10814, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->10da)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > [14:04][ndenev@macbook:~]% > > > > > > I'll try the IPFW workaround, I'm not seeing this at the moment (looks > like > > it was related to how many active torrents there were, as now there are > > none). > > > > --Nikolay > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > No, no PR yet, but I will file one. I wanted to collect some more data first. So, I've did some dtrace digging : [20:54][root@nas:~]#cat reset.d #!/usr/sbin/dtrace -s fbt:kernel:tcp_dropwithreset:entry { printf("reason %d fib %d src_port %d dst_port %d", args[4], args[2] ? args[2]->t_inpcb->inp_inc.inc_fibnum : -1, ntohs(args[1]->th_sport), ntohs(args[1]->th_dport)); /* stack(); */ } [20:54][root@nas:~]#./reset.d dtrace: script './reset.d' matched 1 probe CPU ID FUNCTION:NAME 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 51413 dst_port 47011 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 6881 dst_port 47080 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 16881 dst_port 47009 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 36269 dst_port 47090 0 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 16881 dst_port 47092 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 53263 dst_port 46960 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 22658 dst_port 47079 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 54158 dst_port 47070 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 59547 dst_port 46980 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 16881 dst_port 47092 ^C 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 27890 dst_port 47100 0 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 51413 dst_port 47099 [20:54][root@nas:~]# The port numbers here match RST packets that I'm seeing with tcpdump in another window. reason 3 is BANDLIM_RST_CLOSEDPORT (from icmp_var.h) Looking at tcp_input.c I see that there are cases where the INPCB does not exists, and from what I see this is how the FIB gets determined. Also here I see that tcp_dropwithreset() is called with tcpcb set to NULL, so probably this is why the FIB is not found. Why this is happening, I have no idea yet. --Nikolay
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BP_MZGa8s9PzhoA=kY8YRcG5kRsVb=oP8pjtOXV%2BDH_meG2nQ>