From owner-freebsd-net@FreeBSD.ORG Mon Dec 29 21:03:41 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F12AE1AD; Mon, 29 Dec 2014 21:03:40 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 58E832188; Mon, 29 Dec 2014 21:03:40 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id n3so23027385wiv.7; Mon, 29 Dec 2014 13:03:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=WbUraqnLVjV4AruQPzLzN+bD3hNwU2gz6jYCrvHkEWY=; b=CCogxt9r28SRJpna+ND5tD0PjPpx6Bu4eI1IqCHQT0w9Dbfui7HrbCAUCwRRbxRqID LSoYqvqM5ZGJ7dHrsrUbke03sKkdRn9QMf68xcaC/H013XpRhQLER6rnzo4lpyuA87gZ ckqLW/fC5ztJaECf7tteKvoBWfEPigE+vCrGqLr+ZDju512osX+HCqO3VjA6sV0rmYAk Y/BmeupTVUdZZPIk1z0LtDUQoa22wwvJsSoNwc6IAP58vh1911xiCcz6pVdkmJEM4Gee UwM6GeNK6PomoxU+9OY6w0B0IWU04wCQKd/UhodNypyIoRejel+jeSHGMttjwywuFnWB cXsQ== MIME-Version: 1.0 X-Received: by 10.180.76.144 with SMTP id k16mr98285280wiw.3.1419887018836; Mon, 29 Dec 2014 13:03:38 -0800 (PST) Received: by 10.27.177.218 with HTTP; Mon, 29 Dec 2014 13:03:38 -0800 (PST) In-Reply-To: References: <54A0FDD9.4090009@freebsd.org> Date: Mon, 29 Dec 2014 22:03:38 +0100 Message-ID: Subject: Re: setfib and RSTs From: Nikolay Denev To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Dec 2014 21:03:41 -0000 On Mon, Dec 29, 2014 at 7:47 PM, Adrian Chadd wrote: > Have you filed a PR about this? This seems like it's pretty important to > fix. > > (For other non-torrent reasons too; like people wishing to use this > for things like tor browsing.) > > > -adrian > > > On 29 December 2014 at 05:11, Nikolay Denev wrote: > > On Mon, Dec 29, 2014 at 8:08 AM, Julian Elischer > wrote: > > > >> On 12/26/14 10:41 PM, Nikolay Denev wrote: > >> > >>> Hi, > >>> > >>> I have a process (bittorrent client) running in a non-default fib and > >>> using > >>> a VPN for default gateway: > >>> > >>> from /etc/rc.local : > >>> > >>> /usr/sbin/setfib 1 route add $vpn_provider 10.0.0.1 > >>> /usr/sbin/setfib 1 /usr/local/sbin/openvpn --config > >>> /usr/local/etc/openvpn/provider.ovpn > >>> /usr/sbin/setfib 1 /usr/sbin/service transmission onestart > >>> > >>> Then openvpn installs default gateway in fib 1 to point to the tun(4) > >>> interface. > >>> > >>> Stil, I'm seeing RST packets from the bittorrent client process to be > sent > >>> not via the tunnel, but to the default gateway of the lan which seems > >>> wrong. As if when the kernel generates the RST it does not take into > >>> account the FIB of the socket? > >>> > >> it's possible that you are correct. > >> I checked that RST and other generated packets used the FIB for the > >> session if it existed when they are generated, > >> but I don't know what they do when a single unexpected packet > enters..You > >> may need toset the fib in received packets using either the ipfw setfib > >> command or the ifconfig fib command. The RST should then use the same > FIB > >> to respond. > >> > >> let me know.. > >> > >> > >> Any ideas? > >>> _______________________________________________ > >>> freebsd-net@freebsd.org mailing list > >>> http://lists.freebsd.org/mailman/listinfo/freebsd-net > >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >>> > >>> > >>> > >> > > > > The weird thing was that the RST was a response to a SYN-ACK. > > What I saw was the transmission bittorent daemon to try to connect to > some > > peer over tun2 (the openvpn connection), > > so it sends the SYN, the peer responds with SYN-ACK, then I see RST back > to > > the peer, but not over the tun2, but over lagg0 (as originally > mentioned). > > > > Here 10.8.0.31 is the tun2 address, and XX.XX.XX.XX is one of the peers > > that I was able to capture this happening. (both SYN-ACK and RST's are > > retransmitted, as the RSTs get blocked by the pf running on the default > gw > > as it does not have state for them, as the VPN is a running on a host > > inside the network) > > > > # Captured on tun2: > > > > [14:03][ndenev@macbook:~]%tcpdump -vnr 1.pcap host XX.XX.XX.XX > > reading from file 1.pcap, link-type NULL (BSD loopback) > > 20:43:39.623920 IP (tos 0x0, ttl 64, id 12450, offset 0, flags [DF], > proto > > TCP (6), length 60) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [S], cksum 0xace8 > (correct), > > seq 2985341180, win 8192, options [mss 1460,nop,wscale 6,sackOK,TS val > > 515982338 ecr 0], length 0 > > 20:43:39.912558 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x5a3d > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207923992 ecr 515982338,nop,wscale 7], length 0 > > 20:43:41.132751 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x590a > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207924299 ecr 515982338,nop,wscale 7], length 0 > > 20:43:43.141474 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x5716 > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207924799 ecr 515982338,nop,wscale 7], length 0 > > 20:43:47.129657 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x532e > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207925799 ecr 515982338,nop,wscale 7], length 0 > > 20:43:55.133735 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x4b5e > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207927799 ecr 515982338,nop,wscale 7], length 0 > > 20:44:11.134635 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.60633: Flags [S.], cksum 0x3bbe > > (correct), seq 809114142, ack 2985341181, win 28960, options [mss > > 1368,sackOK,TS val 207931799 ecr 515982338,nop,wscale 7], length 0 > > 20:45:11.290740 IP (tos 0x0, ttl 64, id 61675, offset 0, flags [DF], > proto > > TCP (6), length 60) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [S], cksum 0xdbfb > (correct), > > seq 2733189104, win 8192, options [mss 1460,nop,wscale 6,sackOK,TS val > > 516074004 ecr 0], length 0 > > 20:45:11.595504 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.61382: Flags [S.], cksum 0x81f1 > > (correct), seq 2072870049, ack 2733189105, win 28960, options [mss > > 1368,sackOK,TS val 207946912 ecr 516074004,nop,wscale 7], length 0 > > 20:45:12.748574 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.61382: Flags [S.], cksum 0x80cf > > (correct), seq 2072870049, ack 2733189105, win 28960, options [mss > > 1368,sackOK,TS val 207947202 ecr 516074004,nop,wscale 7], length 0 > > 20:45:14.746692 IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto > TCP > > (6), length 60) > > XX.XX.XX.XX.51413 > 10.8.0.31.61382: Flags [S.], cksum 0x7edb > > (correct), seq 2072870049, ack 2733189105, win 28960, options [mss > > 1368,sackOK,TS val 207947702 ecr 516074004,nop,wscale 7], length 0 > > > > # Captured on lagg0: > > > > [14:04][ndenev@macbook:~]%tcpdump -vnr 2.pcap host XX.XX.XX.XX > > reading from file 2.pcap, link-type EN10MB (Ethernet) > > 20:43:39.912570 IP (tos 0x0, ttl 64, id 12524, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->a2c)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:43:41.132761 IP (tos 0x0, ttl 64, id 13085, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->7fb)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:43:43.141485 IP (tos 0x0, ttl 64, id 14702, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->1aa)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:43:47.129669 IP (tos 0x0, ttl 64, id 16915, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->f904)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:43:55.133748 IP (tos 0x0, ttl 64, id 23534, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->df29)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:44:11.134645 IP (tos 0x0, ttl 64, id 30066, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->c5a5)!) > > 10.8.0.31.60633 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x978a), seq 2985341181, win 0, length 0 > > 20:45:11.595514 IP (tos 0x0, ttl 64, id 62112, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->4877)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:12.748583 IP (tos 0x0, ttl 64, id 62741, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->4602)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:14.746703 IP (tos 0x0, ttl 64, id 63874, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->4195)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:18.754657 IP (tos 0x0, ttl 64, id 65318, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->3bf1)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:26.759522 IP (tos 0x0, ttl 64, id 3473, offset 0, flags [DF], proto > > TCP (6), length 40, bad cksum 0 (->2d87)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > 20:45:42.758524 IP (tos 0x0, ttl 64, id 10814, offset 0, flags [DF], > proto > > TCP (6), length 40, bad cksum 0 (->10da)!) > > 10.8.0.31.61382 > XX.XX.XX.XX.51413: Flags [R], cksum 0xffd2 > (incorrect > > -> 0x2cb1), seq 2733189105, win 0, length 0 > > [14:04][ndenev@macbook:~]% > > > > > > I'll try the IPFW workaround, I'm not seeing this at the moment (looks > like > > it was related to how many active torrents there were, as now there are > > none). > > > > --Nikolay > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > No, no PR yet, but I will file one. I wanted to collect some more data first. So, I've did some dtrace digging : [20:54][root@nas:~]#cat reset.d #!/usr/sbin/dtrace -s fbt:kernel:tcp_dropwithreset:entry { printf("reason %d fib %d src_port %d dst_port %d", args[4], args[2] ? args[2]->t_inpcb->inp_inc.inc_fibnum : -1, ntohs(args[1]->th_sport), ntohs(args[1]->th_dport)); /* stack(); */ } [20:54][root@nas:~]#./reset.d dtrace: script './reset.d' matched 1 probe CPU ID FUNCTION:NAME 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 51413 dst_port 47011 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 6881 dst_port 47080 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 16881 dst_port 47009 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 36269 dst_port 47090 0 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 16881 dst_port 47092 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 53263 dst_port 46960 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 22658 dst_port 47079 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 54158 dst_port 47070 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 59547 dst_port 46980 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 16881 dst_port 47092 ^C 1 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 27890 dst_port 47100 0 7849 tcp_dropwithreset:entry reason 3 fib -1 src_port 51413 dst_port 47099 [20:54][root@nas:~]# The port numbers here match RST packets that I'm seeing with tcpdump in another window. reason 3 is BANDLIM_RST_CLOSEDPORT (from icmp_var.h) Looking at tcp_input.c I see that there are cases where the INPCB does not exists, and from what I see this is how the FIB gets determined. Also here I see that tcp_dropwithreset() is called with tcpcb set to NULL, so probably this is why the FIB is not found. Why this is happening, I have no idea yet. --Nikolay