Date: Fri, 14 May 2010 14:56:12 -0400 From: list@cykotix.com To: freebsd-net@freebsd.org Subject: Packet Loss on FW1 but not FW2 (CARP + PF on FBSD8) Message-ID: <20100514145612.14566x4tj40yhyos@webmail.lahni.com>
next in thread | raw e-mail | index | archive | help
Hello,
I recently just purchased 2 Soekris5501 with identical 120gb 2.5" WD
Scorpio HDDs. I'm using them for network failover, using CARP, PF and
pfSync on FreeBSD 8-STABLE.
The short version of my problem:
I setup FW2 first, imaged its hard drive to FW1. I changed the
necessary configs to update the IPs and ensure FW1 was carp MASTER.
Using a known working port on the switch, I continue to get 70% packet
loss on FW1 on vr0 (vr0 - extif, vr1 - intif, vr2 - pfsync). If I
flip FW1 and FW2, the packet loss follows FW1. I took FW1 home,
plugged it into my home network on vr0 and it works fine with 0%
packet loss so the interface seems fine. I also took the IP bound to
vr0 on FW1 and bound it to vr0 on FW2 and the ISP isn't the problem.
The long version:
Both Soekris5501's use vr0 (ext), vr1 (int) and vr2 (pfsync). I was
given 98.xxx.xxx.58 - .62 with .57 being the gateway IP. FW1 was
assigned .59. FW2 was assigned .60 and I was going to use .58 to NAT
the office traffic over CARP. If I take carp0 and carp1 down off FW1,
it moves all traffic to FW2 appropriately. If I bring carp0 and carp1
back up on FW1, it assumes MASTER again as it should.
FW1 /etc/rc.conf:
-----------------
cloned_interfaces="carp0 carp1"
ifconfig_vr0="inet 98.xxx.xxx.59 netmask 255.255.255.248"
ifconfig_vr1="inet 192.168.1.10 netmask 255.255.255.0"
ifconfig_vr2="inet 10.0.10.12 netmask 255.255.255.0"
ifconfig_carp0="inet 98.xxx.xxx.58 netmask 255.255.255.248 pass
pabsoekris1959 vhid 1"
ifconfig_carp0_alias0="inet 98.xxx.xxx.61 netmask 255.255.255.248"
ifconfig_carp0_alias1="inet 98.xxx.xxx.62 netmask 255.255.255.248"
ifconfig_carp1="inet 192.168.1.1 netmask 255.255.255.0 pass
pabsoekris1959 vhid 2"
ifconfig_pfsync0="syncpeer 10.0.10.13 syncdev vr2"
defaultrouter="98.xxx.xxx.57"
gateway_enable="YES"
FW2 /etc/rc.conf:
-----------------
cloned_interfaces="carp0 carp1"
ifconfig_vr0="inet 98.xxx.xxx.60 netmask 255.255.255.248"
ifconfig_vr1="inet 192.168.1.11 netmask 255.255.255.0"
ifconfig_vr2="inet 10.0.10.13 netmask 255.255.255.0"
ifconfig_carp0="inet 98.xxx.xxx.58 netmask 255.255.255.248 pass
pabsoekris1959 advskew 100 vhid 1"
ifconfig_carp0_alias0="inet 98.xxx.xxx.61 netmask 255.255.255.248"
ifconfig_carp0_alias1="inet 98.xxx.xxx.62 netmask 255.255.255.248"
ifconfig_carp1="inet 192.168.1.1 netmask 255.255.255.0 pass
pabsoekris1959 vhid 2"
ifconfig_pfsync0="syncpeer 10.0.10.12 syncdev vr2"
defaultrouter="98.xxx.xxx.57"
gateway_enable="YES"
FW1 /etc/pf.conf:
------------------------------------------------
ext_if = vr0 # External WAN interface
int_if = vr1 # Internal LAN interface
pfs_if = vr2 # Pfsync interface
carp_extif = carp0 # External CARP interface
carp_intif = carp1
### hosts
office = "192.168.1.0/24"
office_ext = "98.xxx.xxx.58"
soekris1 = "98.xxx.xxx.59"
soekris2 = "98.xxx.xxx.60"
pab = "192.168.1.2"
### icmp
icmp_types = "{ echoreq, unreach }"
### tables
table <bruteforce-ssh> persist
table <badguys> persist file "/etc/badguys"
table <goodguys> { $office }
set block-policy drop
set loginterface $ext_if
set skip on lo
scrub on $ext_if reassemble tcp no-df random-id
### NAT outgoing connections
nat on $ext_if inet from $int_if:network to any -> $office_ext
### port forwards
rdr on $ext_if proto tcp from any to $office_ext port XXXXX -> $pab port 22
rdr on $ext_if proto tcp from any to $office_ext port XXXXX -> $pab port 3389
### ruleset
block in log all # default deny
block in log quick from urpf-failed # spoofed address protection
block in log quick from { <bruteforce-ssh>, <badguys> }
pass log from { lo0, $int_if:network, $ext_if, $carp_extif,
$carp_intif } to any keep state
pass in quick from <goodguys> keep state
pass log inet proto icmp all icmp-type $icmp_types
pass quick on $pfs_if proto pfsync keep state (no-sync) #
enable pfsync
pass on { $int_if, $ext_if } proto carp keep state (no-sync) # enable CARP
FW2 /etc/pf.conf:
-----------------
ext_if = vr0 # External WAN interface
int_if = vr1 # Internal LAN interface
pfs_if = vr2 # Pfsync interface
carp_extif = carp0 # External CARP interface
carp_intif = carp1
### hosts
office = "192.168.1.0/24"
office_ext = "98.xxx.xxx.58"
soekris1 = "98.xxx.xxx.59"
soekris2 = "98.xxx.xxx.60"
pab = "192.168.1.2"
### icmp
icmp_types = "{ echoreq, unreach }"
### tables
table <bruteforce-ssh> persist
table <badguys> persist file "/etc/badguys"
table <goodguys> { $office }
set block-policy drop
set loginterface $ext_if
set skip on lo
scrub on $ext_if reassemble tcp no-df random-id
### NAT outgoing connections
nat on $ext_if inet from $int_if:network to any -> $office_ext
### port forwards
rdr on $ext_if proto tcp from any to $office_ext port XXXXX -> $pab port 22
rdr on $ext_if proto tcp from any to $office_ext port XXXXX -> $pab port 3389
### ruleset
block in log all # default deny
block in log quick from urpf-failed # spoofed address protection
block in log quick from { <bruteforce-ssh>, <badguys> }
pass log from { lo0, $int_if:network, $ext_if, $carp_extif,
$carp_intif } to any keep state
pass in quick from <goodguys> keep state
pass log inet proto icmp all icmp-type $icmp_types
pass quick on $pfs_if proto pfsync keep state (no-sync) #
enable pfsync
pass on { $int_if, $ext_if } proto carp keep state (no-sync) # enable CARP
FW1 ifconfig (carp0 and carp1 are down, packet loss happens regardless):
------------------------------------------------------------------------
soekris1# ifconfig
vr0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
ether 00:00:24:cc:cb:94
inet 98.xxx.xxx.59 netmask 0xfffffff8 broadcast 98.xxx.xxx.63
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vr1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
ether 00:00:24:cc:cb:95
inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vr2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
ether 00:00:24:cc:cb:96
inet 10.0.10.12 netmask 0xffffff00 broadcast 10.0.10.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
pfsync: syncdev: vr2 syncpeer: 10.0.10.13 maxupd: 128
carp0: flags=8<LOOPBACK> metric 0 mtu 1500
inet 98.xxx.xxx.61 netmask 0xfffffff8
inet 98.xxx.xxx.62 netmask 0xfffffff8
inet 98.xxx.xxx.58 netmask 0xfffffff8
carp: INIT vhid 1 advbase 1 advskew 0
carp1: flags=8<LOOPBACK> metric 0 mtu 1500
inet 192.168.1.1 netmask 0xffffff00
carp: INIT vhid 2 advbase 1 advskew 0
FW2 ifconfig (carp0 and carp1 are up and in failover mode):
-----------------------------------------------------------
soekris2# ifconfig
vr0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
ether 00:00:24:ca:40:60
inet 98.xxx.xxx.60 netmask 0xfffffff8 broadcast 98.xxx.xxx.63
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vr1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
ether 00:00:24:ca:40:61
inet 192.168.1.11 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vr2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=280b<RXCSUM,TXCSUM,VLAN_MTU,WOL_UCAST,WOL_MAGIC>
ether 00:00:24:ca:40:62
inet 10.0.10.13 netmask 0xffffff00 broadcast 10.0.10.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1460
pfsync: syncdev: vr2 syncpeer: 10.0.10.12 maxupd: 128
carp0: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
inet 98.xxx.xxx.61 netmask 0xfffffff8
inet 98.xxx.xxx.62 netmask 0xfffffff8
inet 98.xxx.xxx.58 netmask 0xfffffff8
carp: MASTER vhid 1 advbase 1 advskew 100
carp1: flags=49<UP,LOOPBACK,RUNNING> metric 0 mtu 1500
inet 192.168.1.1 netmask 0xffffff00
carp: MASTER vhid 2 advbase 1 advskew 100
Regardless if I flip IPs, flip ports on the switch, anything plugged
into vr0 on FW1 at the office causes 70% packet loss, yet it's fine on
FW2. FW1 vr0 works fine at my house using one of my localnet IPs.
Any suggestions on how to track down where this packet loss is coming
from? I appreciate any input!
Thanks!
Patrick
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100514145612.14566x4tj40yhyos>