Date: Wed, 13 May 2009 20:20:04 GMT From: "Seth Mos" <seth.mos@xs4all.nl> To: freebsd-net@FreeBSD.org Subject: kern/127528: [icmp]: icmp socket receives icmp replies not owned by the process. Message-ID: <200905132020.n4DKK4h1009869@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/127528; it has been noted by GNATS.
From: "Seth Mos" <seth.mos@xs4all.nl>
To: bug-followup@FreeBSD.org, seth.mos@xs4all.nl
Cc:
Subject: kern/127528: [icmp]: icmp socket receives icmp replies not owned
by the process.
Date: Wed, 13 May 2009 21:58:40 +0200 (CEST)
Hi there again, it's been a while.
We are currently on a pfSense 1.2.3-RC1 release and we are now getting a
host of people on the forum and in the mailing lists that are complaining
about the load balancer flapping.
Some debugging turned out some very weird behaviour.
Every now and then, depending on the concurrenty of icmp traffic
originating from the FreeBSD host, ping will miss icmp replies.
Here are the steps performed to debug this issue.
For example the output
tcpdump -ni vlan0 -t icmp
Where vlan0 is the external interface. The IP address being pinged is a
local gateway connected by ethernet so there is nothing but the switch in
between.
http://pastebin.com/f6608c875
The ping command issued is
/sbin/ping -t 4 -oqc 5 -i 0.7 213.23.199.249
This command line will ping the target and exit with a return status of 0
when it receives a reply which can be any of the 5 attempts within the
maximum of 4 seconds.
The output clearly shows that all attempts have unique process IDs, which
is good. It also shows that all attempt have a sequence number of 0 which
is the 1st attempt to ping.
There is however one attempt where this is different, on attempt with pid
46060 you will see that it attempted 4 times to ping the target, it got
valid responses to all 5 requests!.
However, ping exited with a return status of 1 which would imply a timeout.
So now it is worse then the originally reported issue we had on 7.0. We
can now make the FreeBSD distributed ping fail.
Specifically, it fails to see the valid replies.
For your reference I also pasted the tcpdump below. I can also reproduce
this exact same behaviour by using fping. So I would not consider this a
bug in the ping binary itself.
Since the original PR had this issue with apinger, which is different from
both ping and fping I am quite confident there is a serious regression in
here.
# uname -a
FreeBSD noord.coltex.nl 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4 #0: Tue Mar
24 01:18:19 EDT 2009
sullrich@RELENG_1_2-snapshots.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.7
i386
Kind regards,
Seth Mos
pfSense Developer
--------------
1.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 25068, seq
0, length 64
2.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 25068, seq
0, length 64
3.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 27372, seq
0, length 64
4.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 27372, seq
0, length 64
5.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 29676, seq
0, length 64
6.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 29676, seq
0, length 64
7.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 31468, seq
0, length 64
8.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 31468, seq
0, length 64
9.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 33772, seq
0, length 64
10.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 33772, seq
0, length 64
11.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 36076, seq
0, length 64
12.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 36076, seq
0, length 64
13.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
0, length 64
14.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
0, length 64
15.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
1, length 64
16.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
1, length 64
17.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 12525, seq
0, length 64
18.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 12525, seq
0, length 64
19.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 14829, seq
0, length 64
20.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 14829, seq
0, length 64
21.
IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.120.254
unreachable - admin prohibited filter, length 60
22.
IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.148.1
unreachable - admin prohibited filter, length 60
23.
IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.112.1
unreachable - admin prohibited filter, length 60
24.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
2, length 64
25.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
2, length 64
26.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 17645, seq
0, length 64
27.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 17645, seq
0, length 64
28.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
3, length 64
29.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
3, length 64
30.
IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
4, length 64
31.
IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
4, length 64
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200905132020.n4DKK4h1009869>