Date: Tue, 14 Jan 2014 01:24:35 -0600 From: Matthew Grooms <mgrooms@shrew.net> To: freebsd-net@freebsd.org Subject: TCP socket handling lo[X], buggy in 9.x? Message-ID: <52D4E633.1060000@shrew.net>
next in thread | raw e-mail | index | archive | help
All, I am seeing weird issues with tcp connections when they are established from an address that is assigned to a lo interface on 9.2-RELEASE. This works fine on older FreeBSD hosts that I have tested ( like 7.1-RELEASE and 8.2-RELEASE ) but appears to be broken in 9.2-RELEASE-p2. Here is an example of the issue ... host1 ( 9.2-RELEASE ) \ em0: inet 10.22.200.101/24 lo1: inet inet 172.16.1.1/32 host2 ( 8.2-RELEASE ) \ em0: inet 10.22.200.102/24 lo1: inet inet 172.18.1.1/32 ... I add a route on host1 to reach 172.18.1.1/32 on host2 ... [host1]# route add -host 172.18.1.1 10.22.200.102 ... and add a route on host2 to reach 172.16.1.1/32 on host1 ... [host2]# route add -host 172.16.1.1 10.22.200.101 ... Then I use netcat to connect from host1 to host2. When I connect from the local em0 interface to the remote em0 interface, everything is good ... [host1]# nc -s 10.22.200.101 10.22.200.102 1000 line1 line2 line3 line4 [host2]# nc -k -l 1000 line1 line2 line3 line4 ... when I connect from the local em0 interface to the remote lo1 interface, everything looks good ( to lo1 on the 8.2 host ) ... [host1]# nc -s 10.22.200.101 172.18.1.1 1000 line1 line2 line3 line4 [host2]# nc -k -l 1000 line1 line2 line3 line4 ... When I connect from the local lo1 interface to the remote em0 interface, weird things start to happen ( from lo1 on the 9.2 host ) ... [host1]# nc -s 172.16.1.1 10.22.200.102 1000 line1 line2 <connection spontaneously closes> [host1]# nc -s 172.16.1.1 10.22.200.102 1000 line1 line2 <connection spontaneously closes> [host1]# nc -s 172.16.1.1 10.22.200.102 1000 line1 line2 <connection spontaneously closes> [host2]# nc -k -l 1000 line1 line1 line1 ... Here is a comparison of the tcpdump output from scenario #1 vs scenario #3. ... tcpdump -ni em0 src or dst 10.22.200.102 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes 00:49:34.897822 IP 10.22.200.101.18401 > 10.22.200.102.1000: Flags [S], seq 2109171603, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 113771252 ecr 0], length 0 00:49:34.898133 IP 10.22.200.102.1000 > 10.22.200.101.18401: Flags [S.], seq 1520138341, ack 2109171604, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 763106128 ecr 113771252], length 0 00:49:34.898242 IP 10.22.200.101.18401 > 10.22.200.102.1000: Flags [.], ack 1, win 1040, options [nop,nop,TS val 113771252 ecr 763106128], length 0 00:49:36.411982 IP 10.22.200.101.18401 > 10.22.200.102.1000: Flags [P.], seq 1:7, ack 1, win 1040, options [nop,nop,TS val 113772775 ecr 763106128], length 6 00:49:36.502874 IP 10.22.200.102.1000 > 10.22.200.101.18401: Flags [.], ack 7, win 8326, options [nop,nop,TS val 763106289 ecr 113772775], length 0 00:49:37.635691 IP 10.22.200.101.18401 > 10.22.200.102.1000: Flags [P.], seq 7:13, ack 1, win 1040, options [nop,nop,TS val 113773992 ecr 763106289], length 6 00:49:37.733039 IP 10.22.200.102.1000 > 10.22.200.101.18401: Flags [.], ack 13, win 8326, options [nop,nop,TS val 763106412 ecr 113773992], length 0 00:49:40.467889 IP 10.22.200.101.18401 > 10.22.200.102.1000: Flags [P.], seq 13:19, ack 1, win 1040, options [nop,nop,TS val 113776831 ecr 763106412], length 6 00:49:40.562215 IP 10.22.200.102.1000 > 10.22.200.101.18401: Flags [.], ack 19, win 8326, options [nop,nop,TS val 763106695 ecr 113776831], length 0 00:49:42.148281 IP 10.22.200.101.18401 > 10.22.200.102.1000: Flags [P.], seq 19:25, ack 1, win 1040, options [nop,nop,TS val 113778511 ecr 763106695], length 6 00:49:42.242122 IP 10.22.200.102.1000 > 10.22.200.101.18401: Flags [.], ack 25, win 8326, options [nop,nop,TS val 763106863 ecr 113778511], length 0 00:49:42.556244 IP 10.22.200.101.18401 > 10.22.200.102.1000: Flags [F.], seq 25, ack 1, win 1040, options [nop,nop,TS val 113778912 ecr 763106863], length 0 00:49:42.556697 IP 10.22.200.102.1000 > 10.22.200.101.18401: Flags [.], ack 26, win 8326, options [nop,nop,TS val 763106894 ecr 113778912], length 0 00:49:42.556722 IP 10.22.200.102.1000 > 10.22.200.101.18401: Flags [F.], seq 1, ack 26, win 8326, options [nop,nop,TS val 763106894 ecr 113778912], length 0 00:49:42.556806 IP 10.22.200.101.18401 > 10.22.200.102.1000: Flags [.], ack 2, win 1040, options [nop,nop,TS val 113778912 ecr 763106894], length 0 tcpdump -ni em0 src or dst 10.22.200.102 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes 00:52:02.358683 IP 172.16.1.1.10364 > 10.22.200.102.1000: Flags [S], seq 2289590664, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 113918712 ecr 0], length 0 00:52:02.358896 IP 10.22.200.102.1000 > 172.16.1.1.10364: Flags [S.], seq 2860805865, ack 2289590665, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 2333279786 ecr 113918712], length 0 00:52:02.359010 IP 172.16.1.1.10364 > 10.22.200.102.1000: Flags [.], ack 1, win 1040, options [nop,nop,TS val 113918722 ecr 2333279786], length 0 00:52:02.359015 IP 172.16.1.1.10364 > 10.22.200.102.1000: Flags [R], seq 2289590665, win 0, length 0 00:52:04.224632 IP 172.16.1.1.10364 > 10.22.200.102.1000: Flags [P.], seq 1:7, ack 1, win 1040, options [nop,nop,TS val 113920582 ecr 2333279786], length 6 00:52:04.322995 IP 10.22.200.102.1000 > 172.16.1.1.10364: Flags [.], ack 7, win 8326, options [nop,nop,TS val 2333279983 ecr 113920582], length 0 00:52:04.323026 IP 172.16.1.1.10364 > 10.22.200.102.1000: Flags [R], seq 2289590671, win 0, length 0 00:52:05.752341 IP 172.16.1.1.10364 > 10.22.200.102.1000: Flags [P.], seq 7:13, ack 1, win 1040, options [nop,nop,TS val 113922112 ecr 2333279983], length 6 00:52:05.752595 IP 10.22.200.102.1000 > 172.16.1.1.10364: Flags [R], seq 2860805866, win 0, length 0 00:52:06.798407 IP 172.16.1.1.61952 > 10.22.200.102.1000: Flags [S], seq 3819912610, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 113923152 ecr 0], length 0 00:52:06.798588 IP 10.22.200.102.1000 > 172.16.1.1.61952: Flags [S.], seq 889829473, ack 3819912611, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 1369790257 ecr 113923152], length 0 00:52:06.798657 IP 172.16.1.1.61952 > 10.22.200.102.1000: Flags [.], ack 1, win 1040, options [nop,nop,TS val 113923152 ecr 1369790257], length 0 00:52:06.798727 IP 172.16.1.1.61952 > 10.22.200.102.1000: Flags [R], seq 3819912611, win 0, length 0 00:52:08.328904 IP 172.16.1.1.61952 > 10.22.200.102.1000: Flags [P.], seq 1:7, ack 1, win 1040, options [nop,nop,TS val 113924692 ecr 1369790257], length 6 00:52:08.422462 IP 10.22.200.102.1000 > 172.16.1.1.61952: Flags [.], ack 7, win 8326, options [nop,nop,TS val 1369790267 ecr 113924692], length 0 00:52:08.422542 IP 172.16.1.1.61952 > 10.22.200.102.1000: Flags [R], seq 3819912617, win 0, length 0 00:52:09.968562 IP 172.16.1.1.61952 > 10.22.200.102.1000: Flags [P.], seq 7:13, ack 1, win 1040, options [nop,nop,TS val 113926332 ecr 1369790267], length 6 00:52:09.968846 IP 10.22.200.102.1000 > 172.16.1.1.61952: Flags [R], seq 889829474, win 0, length 0 00:52:12.869940 IP 172.16.1.1.35530 > 10.22.200.102.1000: Flags [S], seq 197894496, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 113929232 ecr 0], length 0 00:52:12.870144 IP 10.22.200.102.1000 > 172.16.1.1.35530: Flags [S.], seq 4171403115, ack 197894497, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 2442049100 ecr 113929232], length 0 00:52:12.870211 IP 172.16.1.1.35530 > 10.22.200.102.1000: Flags [.], ack 1, win 1040, options [nop,nop,TS val 113929232 ecr 2442049100], length 0 00:52:12.870331 IP 172.16.1.1.35530 > 10.22.200.102.1000: Flags [R], seq 197894497, win 0, length 0 00:52:14.144771 IP 172.16.1.1.35530 > 10.22.200.102.1000: Flags [P.], seq 1:7, ack 1, win 1040, options [nop,nop,TS val 113930508 ecr 2442049100], length 6 00:52:14.241838 IP 10.22.200.102.1000 > 172.16.1.1.35530: Flags [.], ack 7, win 8326, options [nop,nop,TS val 2442049110 ecr 113930508], length 0 00:52:14.241864 IP 172.16.1.1.35530 > 10.22.200.102.1000: Flags [R], seq 197894503, win 0, length 0 00:52:15.152672 IP 172.16.1.1.35530 > 10.22.200.102.1000: Flags [P.], seq 7:13, ack 1, win 1040, options [nop,nop,TS val 113931512 ecr 2442049110], length 6 00:52:15.153065 IP 10.22.200.102.1000 > 172.16.1.1.35530: Flags [R], seq 4171403116, win 0, length 0 It looks like the 9.2 host is sending a reset before sending every PDU. Weird. In any case, this is really easy to re-produce. Anyone have an idea as to why this started happening? I didn't try FreeBSD 10, but I suspect the problem is the same. Thanks, -Matthew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52D4E633.1060000>