Date: Thu, 4 May 2006 20:03:55 +0400 From: "Dmitry Andrianov" <dimas@dataart.com> To: "Dmitry Andrianov" <dimas@dataart.com>, <freebsd-pf@freebsd.org> Subject: RE: IPSEC tunnel problem Message-ID: <D5972F49810A69449A9EA72A4B360DC2D0A0A9@e1.universe.dart.spb>
next in thread | raw e-mail | index | archive | help
Ok guys,
finally, problem appears to be PF's problem, not an IPSEC one.
We managed to build a test lab and reproduce failures.
=20
At the moment we have:
=20
+--------------------+
| |
| terminal server | Windows box
| 10.2.0.2 |
+--------------------+
|
|
+--------------------+
| fxp1 10.2.0.1 |
| gif0 | FreeBSD 6.0 <--- THIS BOX HAS PROBLEMS
| fxp0 10.1.0.2 |
+--------------------+
|
ipsec goes here
|
+--------------------+
| fxp1 10.1.0.1 |
| gif0 | FreeBSD 6.0
| fxp0 192.168.10.71 |
+--------------------+
|
|
+--------------------+
| 192.168.10.100 |
| Remote desktop | Windows box
| client |
+--------------------+
=20
Everyting below applies to the FreeBSD box marked as "THIS BOX HAS
PROBLEMS" on the "picture" above.
=20
gif0: flags=3D8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
tunnel inet 10.1.0.2 --> 10.1.0.1
inet6 fe80::203:47ff:fe08:3fa6%gif0 prefixlen 64 scopeid 0x5=20
inet 10.2.0.1 --> 192.168.10.71 netmask 0xffffff00
=20
=20
when client connects to Remote Desktop server everything works for some
time and then hangs. On the fxp1 of the box I see:
=20
...
19:52:53.658958 IP 10.2.0.2.3389 > 192.168.10.100.1919: .
62480:63446(966) ack 27922 win 64808
19:52:53.659790 IP 10.2.0.2.3389 > 192.168.10.100.1919: P
63446:64115(669) ack 27922 win 64808
19:52:53.661247 IP 192.168.10.100.1919 > 10.2.0.2.3389: . ack 64115 win
65535
19:52:53.768253 IP 10.2.0.2.3389 > 192.168.10.100.1919: .
64115:65081(966) ack 27922 win 64808
19:52:53.769090 IP 10.2.0.2.3389 > 192.168.10.100.1919: P
65081:65936(855) ack 27922 win 64808
19:52:53.769094 IP 10.2.0.2.3389 > 192.168.10.100.1919: .
65936:66902(966) ack 27922 win 64808
19:52:53.769097 IP 10.2.0.2.3389 > 192.168.10.100.1919: P
66902:67534(632) ack 27922 win 64808
19:52:53.769100 IP 10.2.0.2.3389 > 192.168.10.100.1919: .
67534:68500(966) ack 27922 win 64808
19:52:53.769103 IP 10.2.0.2.3389 > 192.168.10.100.1919: P
68500:68793(293) ack 27922 win 64808
19:52:53.771737 IP 10.2.0.1 > 10.2.0.2: ICMP host 192.168.10.100
unreachable, length 36
19:52:53.772730 IP 10.2.0.1 > 10.2.0.2: ICMP host 192.168.10.100
unreachable, length 36
19:52:53.773724 IP 10.2.0.1 > 10.2.0.2: ICMP host 192.168.10.100
unreachable, length 36
=20
Note that at some moment box starts rejecting packets replying with ICMP
host unreach.
Below is output of pfctl -s info after the test. (Just before the test
started all the stats were reset):
=20
Status: Enabled for 0 days 00:06:00 Debug: Misc
=20
Hostid: 0x1000314c
=20
State Table Total Rate
current entries 3 =20
searches 1387 3.9/s
inserts 3 0.0/s
removals 0 0.0/s
Counters
match 3 0.0/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 0 0.0/s
proto-cksum 0 0.0/s
state-mismatch 13 0.0/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 0 0.0/s
synproxy 0 0.0/s
=20
=20
Note the state-mismatch counter. I have turned on debug for pf (pfctl -x
misc) and grabbed its output during that test. It outputs alot of
"kernel: pf: loose state match: TCP 10.2.0.2:3389 10.2.0.2:3389" which I
assume are normal. But then the following appears:
=20
May 4 19:52:53 vrn1 kernel: pf: BAD state: TCP 10.2.0.2:3389
10.2.0.2:3389 192.168.10.100:1919 [lo=3D4162748520 high=3D4162681620
win=3D65535 modulator=3D0] [lo=3D0 high=3D65535 win=3D1 modulator=3D0] =
2:0 PA
seq=3D4162748520 ack=3D0 len=3D632 ackskew=3D0 pkts=3D245:0 =
dir=3Dout,fwd
May 4 19:52:53 vrn1 kernel: pf: State failure on: 1 | 5 =20
May 4 19:52:53 vrn1 kernel: pf: BAD state: TCP 10.2.0.2:3389
10.2.0.2:3389 192.168.10.100:1919 [lo=3D4162748520 high=3D4162681620
win=3D65535 modulator=3D0] [lo=3D0 high=3D65535 win=3D1 modulator=3D0] =
2:0 A
seq=3D4162749152 ack=3D0 len=3D966 ackskew=3D0 pkts=3D245:0 =
dir=3Dout,fwd
May 4 19:52:53 vrn1 kernel: pf: State failure on: 1 | 5 =20
May 4 19:52:53 vrn1 kernel: pf: BAD state: TCP 10.2.0.2:3389
10.2.0.2:3389 192.168.10.100:1919 [lo=3D4162748520 high=3D4162681620
win=3D65535 modulator=3D0] [lo=3D0 high=3D65535 win=3D1 modulator=3D0] =
2:0 PA
seq=3D4162750118 ack=3D0 len=3D293 ackskew=3D0 pkts=3D245:0 =
dir=3Dout,fwd
May 4 19:52:53 vrn1 kernel: pf: State failure on: 1 | 5=20
=20
I have no idea what this means....
=20
# pfctl -s rules
No ALTQ support in kernel
ALTQ related functions disabled
pass in all keep state
pass out all keep state
=20
If I use "pass in/out all" rules WITHOUT a "keep state" - everything
works fine.
=20
I have tcpdump from all four interfaces (fxp0, fxp1, gif0, pflog0) as
well as full console output. Can send it to any party interested. Also,
for some time this lab will be available for any other tests.
=20
Regards,
Dmitry Andrianov
=20
________________________________
From: Dmitry Andrianov=20
Sent: Friday, April 28, 2006 5:38 PM
To: freebsd-pf@freebsd.org
Subject: IPSEC tunnel problem
Hello.
First of all I apologize if I freebsd-pf is not the rigth place to ask
my question. I will explain below why it is actually asked here. But if
anyone knows the better place, let me know.
[stripped]
=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D5972F49810A69449A9EA72A4B360DC2D0A0A9>