Date: Sun, 23 Oct 2005 15:37:03 +1000 From: Michael VInce <mv@roq.com> To: Volker <volker@vwsoft.com> Cc: Max Laier <max@love2party.net>, freebsd-net@freebsd.org Subject: Re: IPSec tcp session stalling Message-ID: <435B217F.70106@roq.com> In-Reply-To: <435ADFB5.10603@vwsoft.com> References: <435A5D9B.7080309@vwsoft.com> <435A900C.3060602@roq.com> <435AD808.1030701@vwsoft.com> <200510230140.42154.max@love2party.net> <435ADFB5.10603@vwsoft.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I am using FAST_IPSEC on a multi subnet VPN with the guys on other side having Check Point VPN / Firewall. Its a VPN that does almost non stop usage, the people on the other side have 24 monitoring utils on it and its never had a problem. Its on 5.3 i386, and I fear to upgrade it, when it comes to VPN I believe in the rule if its not broke don't fix it. When I think about if I haven't had much luck when trying regular IPSEC despite docs saying its better supported? But then again I never gave it a good shot, FAST_IPSEC just sounded 'faster' Mike Volker wrote: >Max & Co: > >I've just seen I'm using kernel config 'options IPSEC' on both machines. >Should I try 'options FAST_IPSEC'? Would take some hours for kernel >recompile. Does the code IPSEC / FAST_IPSEC make a difference (even >while having not hardware crypto accelerator)? > >May I use FAST_IPSEC even without any hw-crypto devices? While reading >`man fast_ipsec' I would think it depends on a hw-crypto device... > >Please tell me if we should check IPSEC / FAST_IPSEC and I'll start a >recompile. > >Volker > > >On 2005-10-23 00:40, Max Laier wrote: > > >>To try something else: Could you guys try to disable SACK on the machines >>involved? I haven't looked at the dumps as of yet, but that's one simple >>test that might help to identify the problem. >> >>sysctl net.inet.tcp.sack.enable=0 >> >>On Sunday 23 October 2005 02:23, Volker wrote: >> >> >> >>>Michael, >>> >>>I not that sure if I'm right in checking what you suggested but when >>>trying to do ping hostB from hostA with oversized packets through the >>>IPSec tunnel by: >>> >>># ping -c 10 -s 12000 10.128.6.1 >>> >>>I'm getting replies easily. >>> >>>While doing that and tcpdump'ing the gif interface, I'm seeing the >>>fragmented packets coming in properly. >>> >>>If that's a reliable check for MTU than the problem should not be MTU >>>related. Is there any other way to check MTU problems by using `ping'? >>> >>>Thanks, >>> >>>Volker >>> >>>On 2005-10-22 20:16, Michael VInce wrote: >>> >>> >>> >>>>Try sending different sized pings or other packet size control utils to >>>>really make sure its not MTU related. >>>>Maybe there is an upstream router thats blocking ICMP fragment packets, >>>>have you ever seen them? try forcing the creation of some. >>>> >>>>Mike >>>> >>>>Volker wrote: >>>> >>>> >>>> >>>>>Still having the same problem with an IPSec tunnel between FreeBSD 5.4R >>>>>hosts. >>>>> >>>>>Problem description: >>>>>scp session tries to transfer a large file through an IPSec tunnel. The >>>>>file is being transmitted but scp says 'stalled' after 56K (49152 bytes >>>>>file size). The IPSec tunnel itself is still up even after the scp >>>>>abort. Other tcp sessions break, too when sending too much traffic >>>>>through the tunnel. >>>>> >>>>>I've taken a closer look to it and tried to get something useful out of >>>>>the tcpdump but I'm unable to see any errors or I'm misinterpreting >>>>>something. >>>>> >>>>>The connection looks like: >>>>> >>>>>extIP: A.B.C.D >>>>>extIP: E.F.G.H >>>>>host A ------------------ (internet) ------------------ host B >>>>>tunnelIP: 10.128.1.6 tunnelIP: >>>>>10.128.6.1 >>>>> >>>>>host A just has an external interface (em1) connected to a leased line >>>>>with a fixed IP address (IP-addr A.B.C.D). >>>>>host B has an S-DSL connection at xl0, PPPoE at ng0 (IP-addr. E.F.G.H). >>>>> >>>>>Both hosts are using gif for the IPSec tunnel. >>>>> >>>>>The routing tables (netstat -rnWf inet) are looking good and IMHO the >>>>>MTU is fine. >>>>> >>>>>host A: >>>>>em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 >>>>> options=b<RXCSUM,TXCSUM,VLAN_MTU> >>>>> inet A.B.C.D netmask 0xfffffff8 broadcast A.B.C.z >>>>> ether 00:c0:9f:46:ec:c7 >>>>> media: Ethernet autoselect (100baseTX <full-duplex>) >>>>> status: active >>>>>gif6: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280 >>>>> tunnel inet A.B.C.D --> E.F.G.H >>>>> inet 10.128.1.6 --> 10.128.6.1 netmask 0xffffffff >>>>> inet6 fe80::2c0:9fff:fe46:ecc6%gif6 prefixlen 64 scopeid 0x4 >>>>> >>>>>Routing tables (shortened) >>>>>Destination Gateway Flags Refs Use Mtu >>>>>Netif Expire >>>>>default A.B.C.x UGS 2 516686 1500 em1 >>>>>10.128.1.6 127.0.0.1 UH 0 14 >>>>>16384 lo0 >>>>>10.128.6.1 10.128.1.6 UH 0 6017 >>>>>1280 gif6 >>>>>127.0.0.1 127.0.0.1 UH 0 31633 >>>>>16384 lo0 >>>>>A.B.C.x/29 link#2 UC 0 0 1500 em1 >>>>>A.B.C.D 00:c0:9f:46:ec:c7 UHLW 0 112 1500 lo0 >>>>> >>>>>On host B the interfaces and routing tables are looking like: >>>>>xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 >>>>> options=8<VLAN_MTU> >>>>> inet 0.0.0.0 netmask 0xff000000 broadcast 0.255.255.255 >>>>> inet6 fe80::260:8ff:fe6c:e73c%xl0 prefixlen 64 scopeid 0x1 >>>>> ether 00:60:08:6c:e7:3c >>>>> media: Ethernet 10baseT/UTP (10baseT/UTP <half-duplex>) >>>>> status: active >>>>>gif1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280 >>>>> tunnel inet E.F.G.H --> A.B.C.D >>>>> inet6 fe80::260:8ff:fe6c:e73c%gif1 prefixlen 64 scopeid 0x4 >>>>> inet 10.128.6.1 --> 10.128.1.6 netmask 0xffffffff >>>>>ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1456 >>>>> inet E.F.G.H --> 217.5.98.186 netmask 0xffffffff >>>>> >>>>>Routing tables (shortened) >>>>>Destination Gateway Flags Refs Use Mtu >>>>>Netif Expire >>>>>0 link#1 UC 0 0 1500 >>>>>xl0 => >>>>>default 217.5.98.186 UGS 1 38474 >>>>>1456 ng0 >>>>>10.128.1.6 10.128.6.1 UH 4 2196 >>>>>1280 gif1 >>>>>127.0.0.1 127.0.0.1 UH 0 80424 >>>>>16384 lo0 >>>>>217.5.98.186 E.F.G.H UH 1 0 1456 ng0 >>>>>E.F.G.H lo0 UHS 0 0 16384 lo0 >>>>> >>>>>While trying to fetch a file by scp on host A (receiver) from host B >>>>>(sender), I captured the following tcpdump on host B: >>>>> >>>>>tcpdump -netttvvi gif1: >>>>> >>>>> >>>>> >>>>>>000023 AF 2 1280: IP (tos 0x8, ttl 64, id 13202, offset 0, flags >>>>>>[none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>43864:45092(1228) ack 1330 win 33156 <nop,nop,timestamp 481770567 >>>>>>565002838> >>>>>>000207 AF 2 1280: IP (tos 0x8, ttl 64, id 52187, offset 0, flags >>>>>>[none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>45092:46320(1228) ack 1330 win 33156 <nop,nop,timestamp 481770567 >>>>>>565002838> >>>>>>000220 AF 2 1280: IP (tos 0x8, ttl 64, id 33774, offset 0, flags >>>>>>[none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>46320:47548(1228) ack 1330 win 33156 <nop,nop,timestamp 481770568 >>>>>>565002838> >>>>>>003524 AF 2 52: IP (tos 0x8, ttl 64, id 42063, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 38952 win 33156 <nop,nop,timestamp 565002844 >>>>>>481770524> 000024 AF 2 1280: IP (tos 0x8, ttl 64, id 48541, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>47548:48776(1228) ack 1330 win 33156 <nop,nop,timestamp 481770571 >>>>>>565002844> >>>>>>011203 AF 2 52: IP (tos 0x8, ttl 64, id 60517, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 41408 win 32542 <nop,nop,timestamp 565002855 >>>>>>481770530> 000058 AF 2 1280: IP (tos 0x8, ttl 64, id 15798, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>48776:50004(1228) ack 1330 win 33156 <nop,nop,timestamp 481770582 >>>>>>565002855> >>>>>>000246 AF 2 1280: IP (tos 0x8, ttl 64, id 31721, offset 0, flags >>>>>>[none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>50004:51232(1228) ack 1330 win 33156 <nop,nop,timestamp 481770583 >>>>>>565002855> >>>>>>005147 AF 2 52: IP (tos 0x8, ttl 64, id 22347, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 42636 win 33156 <nop,nop,timestamp 565002861 >>>>>>481770542> 000024 AF 2 1280: IP (tos 0x8, ttl 64, id 61057, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>51232:52460(1228) ack 1330 win 33156 <nop,nop,timestamp 481770588 >>>>>>565002861> >>>>>>020769 AF 2 52: IP (tos 0x8, ttl 64, id 27692, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 45092 win 32542 <nop,nop,timestamp 565002881 >>>>>>481770547> 000027 AF 2 1280: IP (tos 0x8, ttl 64, id 64167, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>52460:53688(1228) ack 1330 win 33156 <nop,nop,timestamp 481770609 >>>>>>565002881> >>>>>>000209 AF 2 1280: IP (tos 0x8, ttl 64, id 45457, offset 0, flags >>>>>>[none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>53688:54916(1228) ack 1330 win 33156 <nop,nop,timestamp 481770609 >>>>>>565002881> >>>>>>005260 AF 2 52: IP (tos 0x8, ttl 64, id 53832, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 46320 win 33156 <nop,nop,timestamp 565002887 >>>>>>481770567> 000024 AF 2 1280: IP (tos 0x8, ttl 64, id 3515, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>54916:56144(1228) ack 1330 win 33156 <nop,nop,timestamp 481770614 >>>>>>565002887> >>>>>>011020 AF 2 52: IP (tos 0x8, ttl 64, id 11608, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 48776 win 32542 <nop,nop,timestamp 565002898 >>>>>>481770568> 000026 AF 2 1280: IP (tos 0x8, ttl 64, id 5848, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>56144:57372(1228) ack 1330 win 33156 <nop,nop,timestamp 481770625 >>>>>>565002898> >>>>>>000211 AF 2 1280: IP (tos 0x8, ttl 64, id 39892, offset 0, flags >>>>>>[none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>57372:58600(1228) ack 1330 win 33156 <nop,nop,timestamp 481770625 >>>>>>565002898> >>>>>>005641 AF 2 52: IP (tos 0x8, ttl 64, id 7943, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 50004 win 33156 <nop,nop,timestamp 565002904 >>>>>>481770582> 000024 AF 2 1280: IP (tos 0x8, ttl 64, id 8678, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>58600:59828(1228) ack 1330 win 33156 <nop,nop,timestamp 481770631 >>>>>>565002904> >>>>>>011072 AF 2 52: IP (tos 0x8, ttl 64, id 38257, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 52460 win 32542 <nop,nop,timestamp 565002915 >>>>>>481770583> 000025 AF 2 1280: IP (tos 0x8, ttl 64, id 12255, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>59828:61056(1228) ack 1330 win 33156 <nop,nop,timestamp 481770642 >>>>>>565002915> >>>>>>000209 AF 2 1280: IP (tos 0x8, ttl 64, id 46257, offset 0, flags >>>>>>[none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>61056:62284(1228) ack 1330 win 33156 <nop,nop,timestamp 481770642 >>>>>>565002915> >>>>>>000222 AF 2 1280: IP (tos 0x8, ttl 64, id 4093, offset 0, flags >>>>>>[none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>62284:63512(1228) ack 1330 win 33156 <nop,nop,timestamp 481770643 >>>>>>565002915> >>>>>>007065 AF 2 52: IP (tos 0x8, ttl 64, id 18720, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 53688 win 33156 <nop,nop,timestamp 565002922 >>>>>>481770609> 000025 AF 2 1280: IP (tos 0x8, ttl 64, id 38378, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>63512:64740(1228) ack 1330 win 33156 <nop,nop,timestamp 481770650 >>>>>>565002922> >>>>>>011034 AF 2 52: IP (tos 0x8, ttl 64, id 18718, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 56144 win 32542 <nop,nop,timestamp 565002934 >>>>>>481770609> 000024 AF 2 1280: IP (tos 0x8, ttl 64, id 8148, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.53160: . >>>>>>64740:65968(1228) ack 1330 win 33156 <nop,nop,timestamp 481770661 >>>>>>565002934> >>>>>>005991 AF 2 52: IP (tos 0x8, ttl 64, id 62285, offset 0, flags >>>>>>[none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum ok] >>>>>>1330:1330(0) ack 57372 win 33156 <nop,nop,timestamp 565002939 >>>>>>481770625> 010726 AF 2 52: IP (tos 0x8, ttl 64, id 1549, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum >>>>>>ok] 1330:1330(0) ack 59828 win 32542 <nop,nop,timestamp 565002950 >>>>>>481770625> 005670 AF 2 52: IP (tos 0x8, ttl 64, id 61504, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum >>>>>>ok] 1330:1330(0) ack 61056 win 33156 <nop,nop,timestamp 565002956 >>>>>>481770642> 011260 AF 2 52: IP (tos 0x8, ttl 64, id 32633, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum >>>>>>ok] 1330:1330(0) ack 63512 win 32542 <nop,nop,timestamp 565002967 >>>>>>481770642> 005510 AF 2 52: IP (tos 0x8, ttl 64, id 54614, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum >>>>>>ok] 1330:1330(0) ack 64740 win 33156 <nop,nop,timestamp 565002973 >>>>>>481770650> 104909 AF 2 52: IP (tos 0x8, ttl 64, id 50471, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.53160 > 10.128.6.1.22: . [tcp sum >>>>>>ok] 1330:1330(0) ack 65968 win 33156 <nop,nop,timestamp 565003078 >>>>>>481770661> >>>>>> >>>>>> >>>>>tcpdump -netttvvi ng0 host A.B.C.D: >>>>> >>>>> >>>>> >>>>>>000227 AF 2 1352: IP (tos 0x8, ttl 64, id 25895, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10b) >>>>>>011042 AF 2 128: IP (tos 0x8, ttl 61, id 5786, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf0) >>>>>>000226 AF 2 1352: IP (tos 0x8, ttl 64, id 36701, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10c) >>>>>>000216 AF 2 1352: IP (tos 0x8, ttl 64, id 8789, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10d) >>>>>>004853 AF 2 128: IP (tos 0x8, ttl 61, id 17128, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf1) >>>>>>000227 AF 2 1352: IP (tos 0x8, ttl 64, id 34888, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10e) >>>>>>018747 AF 2 128: IP (tos 0x8, ttl 61, id 14828, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf2) >>>>>>000248 AF 2 1352: IP (tos 0x8, ttl 64, id 34356, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x10f) >>>>>>000223 AF 2 1352: IP (tos 0x8, ttl 64, id 34151, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x110) >>>>>>005030 AF 2 128: IP (tos 0x8, ttl 61, id 45476, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf3) >>>>>>000228 AF 2 1352: IP (tos 0x8, ttl 64, id 39765, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x111) >>>>>>011247 AF 2 128: IP (tos 0x8, ttl 61, id 63692, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf4) >>>>>>000226 AF 2 1352: IP (tos 0x8, ttl 64, id 29240, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x112) >>>>>>000222 AF 2 1352: IP (tos 0x8, ttl 64, id 43306, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x113) >>>>>>005663 AF 2 128: IP (tos 0x8, ttl 61, id 32980, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf5) >>>>>>000228 AF 2 1352: IP (tos 0x8, ttl 64, id 56920, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x114) >>>>>>010190 AF 2 128: IP (tos 0x8, ttl 61, id 3206, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf6) >>>>>>000227 AF 2 1352: IP (tos 0x8, ttl 64, id 4655, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x115) >>>>>>000215 AF 2 1352: IP (tos 0x8, ttl 64, id 62740, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x116) >>>>>>000203 AF 2 1352: IP (tos 0x8, ttl 64, id 35642, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x117) >>>>>>006875 AF 2 128: IP (tos 0x8, ttl 61, id 37801, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf7) >>>>>>000234 AF 2 1352: IP (tos 0x8, ttl 64, id 41803, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x118) >>>>>>010651 AF 2 128: IP (tos 0x8, ttl 61, id 54256, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf8) >>>>>>000235 AF 2 1352: IP (tos 0x8, ttl 64, id 30732, offset 0, flags >>>>>>[none], length: 1352) E.F.G.H > A.B.C.D: ESP(spi=0x078b2968,seq=0x119) >>>>>>007913 AF 2 128: IP (tos 0x8, ttl 61, id 7647, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xf9) >>>>>>011166 AF 2 128: IP (tos 0x8, ttl 61, id 58037, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfa) >>>>>>005483 AF 2 128: IP (tos 0x8, ttl 61, id 65275, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfb) >>>>>>011250 AF 2 128: IP (tos 0x8, ttl 61, id 47289, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfc) >>>>>>005505 AF 2 128: IP (tos 0x8, ttl 61, id 203, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfd) >>>>>>104747 AF 2 128: IP (tos 0x8, ttl 61, id 45263, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xfe) >>>>>>8. 338674 AF 2 128: IP (tos 0x8, ttl 61, id 36351, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0xff) >>>>>>319992 AF 2 128: IP (tos 0x8, ttl 61, id 18085, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x100) >>>>>>441837 AF 2 128: IP (tos 0x8, ttl 61, id 58323, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x101) >>>>>>684077 AF 2 128: IP (tos 0x8, ttl 61, id 35487, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x102) >>>>>>1. 167602 AF 2 128: IP (tos 0x8, ttl 61, id 34442, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x103) >>>>>>2. 136032 AF 2 128: IP (tos 0x8, ttl 61, id 8345, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x104) >>>>>>2. 984665 AF 2 128: IP (tos 0x8, ttl 61, id 35456, offset 0, flags >>>>>>[none], length: 128) A.B.C.D > E.F.G.H: ESP(spi=0x0858046f,seq=0x105) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>From what I'm seeing host B just stops sending without any reason. At >>>>> >>>>>least I don't see any fragmented packets. The only thing I've seen is >>>>>some packets doesn't get ack'ed by the receiver. >>>>> >>>>>These packets never get ack'ed: >>>>>46320:47548(1228) >>>>>50004:51232(1228) >>>>>53688:54916(1228) >>>>>57372:58600(1228) >>>>>61056:62284(1228) >>>>> >>>>>On host A I dumped the following: >>>>> >>>>>tcpdump -netttvvi gif6 >>>>> >>>>> >>>>> >>>>> >>>>>>1129985378.941282 AF 2 52: IP (tos 0x8, ttl 64, id 41637, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 45092 win 32542 <nop,nop,timestamp 574090240 >>>>>>490857876> >>>>>>1129985378.952628 AF 2 1280: IP (tos 0x8, ttl 64, id 14004, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>45092:46320(1228) ack 1330 win 33156 <nop,nop,timestamp 490857901 >>>>>>574090210> >>>>>>1129985378.952657 AF 2 52: IP (tos 0x8, ttl 64, id 23243, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 46320 win 33156 <nop,nop,timestamp 574090251 >>>>>>490857901> >>>>>>1129985378.958250 AF 2 1280: IP (tos 0x8, ttl 64, id 4306, offset 0, >>>>>>flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>46320:47548(1228) ack 1330 win 33156 <nop,nop,timestamp 490857901 >>>>>>574090210> >>>>>>1129985378.971118 AF 2 1280: IP (tos 0x8, ttl 64, id 33534, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>47548:48776(1228) ack 1330 win 33156 <nop,nop,timestamp 490857920 >>>>>>574090229> >>>>>>1129985378.971137 AF 2 52: IP (tos 0x8, ttl 64, id 60095, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 48776 win 32542 <nop,nop,timestamp 574090270 >>>>>>490857901> >>>>>>1129985378.982488 AF 2 1280: IP (tos 0x8, ttl 64, id 11459, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>48776:50004(1228) ack 1330 win 33156 <nop,nop,timestamp 490857931 >>>>>>574090240> >>>>>>1129985378.982516 AF 2 52: IP (tos 0x8, ttl 64, id 33184, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 50004 win 33156 <nop,nop,timestamp 574090281 >>>>>>490857931> >>>>>>1129985378.987989 AF 2 1280: IP (tos 0x8, ttl 64, id 54180, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>50004:51232(1228) ack 1330 win 33156 <nop,nop,timestamp 490857931 >>>>>>574090240> >>>>>>1129985378.994231 AF 2 1280: IP (tos 0x8, ttl 64, id 24535, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>51232:52460(1228) ack 1330 win 33156 <nop,nop,timestamp 490857942 >>>>>>574090251> >>>>>>1129985378.994250 AF 2 52: IP (tos 0x8, ttl 64, id 30647, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 52460 win 32542 <nop,nop,timestamp 574090293 >>>>>>490857931> >>>>>>1129985379.012101 AF 2 1280: IP (tos 0x8, ttl 64, id 61397, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>52460:53688(1228) ack 1330 win 33156 <nop,nop,timestamp 490857960 >>>>>>574090270> >>>>>>1129985379.012132 AF 2 52: IP (tos 0x8, ttl 64, id 60550, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 53688 win 33156 <nop,nop,timestamp 574090311 >>>>>>490857960> >>>>>>1129985379.017754 AF 2 1280: IP (tos 0x8, ttl 64, id 28408, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>53688:54916(1228) ack 1330 win 33156 <nop,nop,timestamp 490857961 >>>>>>574090270> >>>>>>1129985379.023720 AF 2 1280: IP (tos 0x8, ttl 64, id 27558, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>54916:56144(1228) ack 1330 win 33156 <nop,nop,timestamp 490857972 >>>>>>574090281> >>>>>>1129985379.023741 AF 2 52: IP (tos 0x8, ttl 64, id 21502, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 56144 win 32542 <nop,nop,timestamp 574090322 >>>>>>490857961> >>>>>>1129985379.035333 AF 2 1280: IP (tos 0x8, ttl 64, id 18885, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>56144:57372(1228) ack 1330 win 33156 <nop,nop,timestamp 490857984 >>>>>>574090293> >>>>>>1129985379.035362 AF 2 52: IP (tos 0x8, ttl 64, id 59875, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 57372 win 33156 <nop,nop,timestamp 574090334 >>>>>>490857984> >>>>>>1129985379.040830 AF 2 1280: IP (tos 0x8, ttl 64, id 37252, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>57372:58600(1228) ack 1330 win 33156 <nop,nop,timestamp 490857984 >>>>>>574090293> >>>>>>1129985379.046576 AF 2 1280: IP (tos 0x8, ttl 64, id 18349, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>58600:59828(1228) ack 1330 win 33156 <nop,nop,timestamp 490857984 >>>>>>574090293> >>>>>>1129985379.046595 AF 2 52: IP (tos 0x8, ttl 64, id 43697, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 59828 win 32542 <nop,nop,timestamp 574090345 >>>>>>490857984> >>>>>>1129985379.064961 AF 2 1280: IP (tos 0x8, ttl 64, id 38300, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>59828:61056(1228) ack 1330 win 33156 <nop,nop,timestamp 490858013 >>>>>>574090322> >>>>>>1129985379.064993 AF 2 52: IP (tos 0x8, ttl 64, id 47539, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 61056 win 33156 <nop,nop,timestamp 574090364 >>>>>>490858013> >>>>>>1129985379.070688 AF 2 1280: IP (tos 0x8, ttl 64, id 30345, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>61056:62284(1228) ack 1330 win 33156 <nop,nop,timestamp 490858013 >>>>>>574090322> >>>>>>1129985379.076184 AF 2 1280: IP (tos 0x8, ttl 64, id 37536, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>62284:63512(1228) ack 1330 win 33156 <nop,nop,timestamp 490858014 >>>>>>574090322> >>>>>>1129985379.076202 AF 2 52: IP (tos 0x8, ttl 64, id 34201, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 63512 win 32542 <nop,nop,timestamp 574090375 >>>>>>490858013> >>>>>>1129985379.081680 AF 2 1280: IP (tos 0x8, ttl 64, id 20637, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>63512:64740(1228) ack 1330 win 33156 <nop,nop,timestamp 490858025 >>>>>>574090334> >>>>>>1129985379.081709 AF 2 52: IP (tos 0x8, ttl 64, id 59866, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 64740 win 33156 <nop,nop,timestamp 574090380 >>>>>>490858025> >>>>>>1129985379.087678 AF 2 1280: IP (tos 0x8, ttl 64, id 35213, offset >>>>>>0, flags [none], length: 1280) 10.128.6.1.22 > 10.128.1.6.59762: . >>>>>>64740:65968(1228) ack 1330 win 33156 <nop,nop,timestamp 490858036 >>>>>>574090345> >>>>>>1129985379.186906 AF 2 52: IP (tos 0x8, ttl 64, id 2465, offset 0, >>>>>>flags [none], length: 52) 10.128.1.6.59762 > 10.128.6.1.22: . [tcp >>>>>>sum ok] 1330:1330(0) ack 65968 win 33156 <nop,nop,timestamp 574090486 >>>>>>490858036> >>>>>> >>>>>> >>>>>tcpdump -netttvvi em1 host E.F.G.H >>>>> >>>>> >>>>> >>>>> >>>>>>1129985379.064825 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype >>>>>>IPv4 (0x0800), length 1366: IP (tos 0x8, ttl 61, id 45003, offset 0, >>>>>>flags [none], length: 1352) E.F.G.H > A.B.C.D: >>>>>>ESP(spi=0x0e0dffaa,seq=0x3e) >>>>>>1129985379.065024 00:c0:9f:46:ec:c7 > 00:13:c4:fa:6c:20, ethertype >>>>>>IPv4 (0x0800), length 142: IP (tos 0x8, ttl 64, id 1195, offset 0, >>>>>>flags [none], length: 128) A.B.C.D > E.F.G.H: >>>>>>ESP(spi=0x029a41b4,seq=0x2f) >>>>>>1129985379.070572 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype >>>>>>IPv4 (0x0800), length 1366: IP (tos 0x8, ttl 61, id 36820, offset 0, >>>>>>flags [none], length: 1352) E.F.G.H > A.B.C.D: >>>>>>ESP(spi=0x0e0dffaa,seq=0x3f) >>>>>>1129985379.076069 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype >>>>>>IPv4 (0x0800), length 1366: IP (tos 0x8, ttl 61, id 44971, offset 0, >>>>>>flags [none], length: 1352) E.F.G.H > A.B.C.D: >>>>>>ESP(spi=0x0e0dffaa,seq=0x40) >>>>>>1129985379.076233 00:c0:9f:46:ec:c7 > 00:13:c4:fa:6c:20, ethertype >>>>>>IPv4 (0x0800), length 142: IP (tos 0x8, ttl 64, id 56964, offset 0, >>>>>>flags [none], length: 128) A.B.C.D > E.F.G.H: >>>>>>ESP(spi=0x029a41b4,seq=0x30) >>>>>>1129985379.081565 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype >>>>>>IPv4 (0x0800), length 1366: IP (tos 0x8, ttl 61, id 24742, offset 0, >>>>>>flags [none], length: 1352) E.F.G.H > A.B.C.D: >>>>>>ESP(spi=0x0e0dffaa,seq=0x41) >>>>>>1129985379.081741 00:c0:9f:46:ec:c7 > 00:13:c4:fa:6c:20, ethertype >>>>>>IPv4 (0x0800), length 142: IP (tos 0x8, ttl 64, id 9390, offset 0, >>>>>>flags [none], length: 128) A.B.C.D > E.F.G.H: >>>>>>ESP(spi=0x029a41b4,seq=0x31) >>>>>>1129985379.087562 00:13:c4:fa:6c:20 > 00:c0:9f:46:ec:c7, ethertype >>>>>>IPv4 (0x0800), length 1366: IP (tos 0x8, ttl 61, id 48065, offset 0, >>>>>>flags [none], length: 1352) E.F.G.H > A.B.C.D: >>>>>>ESP(spi=0x0e0dffaa,seq=0x42) >>>>>>1129985379.186945 00:c0:9f:46:ec:c7 > 00:13:c4:fa:6c:20, ethertype >>>>>>IPv4 (0x0800), length 142: IP (tos 0x8, ttl 64, id 36315, offset 0, >>>>>>flags [none], length: 128) A.B.C.D > E.F.G.H: >>>>>>ESP(spi=0x029a41b4,seq=0x32) >>>>>> >>>>>> >>>>>If I'm not misleaded, this also doesn't show any errors except the >>>>>missing ack's. host B just stops sending. If there's an ack missing, >>>>>doesn't have the sending host to just repeat the un-ack'ed packet? >>>>> >>>>>The IPSec tunnel does not die. Even shortly after the (scp) transfer >>>>>stalls the tunnel itself is still usable (for small amounts of data). To >>>>>make it more worse, when disabling pf at the senders side, the transfer >>>>>works. I've tripple checked pflog for denied packets on both sides but >>>>>pf didn't filter any packets out. >>>>> >>>>>When disabling the IPSec rules using `setkey -F; setkey -FP' on the >>>>>tunnel for a moment, the scp transfer does not stall. So it's not a gif >>>>>issue. >>>>> >>>>>It doesn't seem to be an MTU issue (pf has also the rule 'scrub in/out >>>>>all no-df'), but what kind of issue is that?? Has anybody ever >>>>>experienced similar things? Or am I misinterpreting the tcpdump output? >>>>> >>>>> >>>>>Any help and hint is appreciated! Without an error message I'm lost. >>>>> >>>>>Volker >>>>> >>>>>_______________________________________________ >>>>>freebsd-net@freebsd.org mailing list >>>>>http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>>>> >>>>> >>>_______________________________________________ >>>freebsd-net@freebsd.org mailing list >>>http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >>> >> >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?435B217F.70106>