From owner-freebsd-net@FreeBSD.ORG Fri Jan 2 13:39:01 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 153B7A07 for ; Fri, 2 Jan 2015 13:39:01 +0000 (UTC) Received: from mail-wg0-x231.google.com (mail-wg0-x231.google.com [IPv6:2a00:1450:400c:c00::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 80ACB66FFF for ; Fri, 2 Jan 2015 13:39:00 +0000 (UTC) Received: by mail-wg0-f49.google.com with SMTP id n12so23971882wgh.8 for ; Fri, 02 Jan 2015 05:38:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=GioyB88Ot8w3nIxNVA+E1ZTsHG3rUGiRkBMZGZommRA=; b=XTdS+2QoHJ0y0iq8i4Fd24ty/9BjseF/VG2iR7X28kyeewNrLhbnzmGnEs6TFTaGBq agIC+H80D0kXacXJ68R+2yxecX2Wr9ob2lbmWcnk2BoMxjqnfAFnrvzSw9Y1R/OL5UeX FMc1mYwcXgJtW4wSoVAnvvDEHKp28hSSSF+0mWMBUN4eH4rzvONQgvW5IlA3Lp7vDTPW eQhOwq1hJHwsIjKjdTDY+9+ZInJEbzRsLWKwSdA3qYSvHU0wBFjtOqZ7Ob66+kzHjZfq nOcg9PupOhRZG5RaA0AHFymGSR7j5B5uBYxi1rt5DIzosqyYxvZqxyTIcAs9axg9QLhg SV1Q== X-Received: by 10.194.236.1 with SMTP id uq1mr132958452wjc.28.1420205937751; Fri, 02 Jan 2015 05:38:57 -0800 (PST) Received: from [10.99.0.3] ([213.188.107.182]) by mx.google.com with ESMTPSA id td6sm54163037wic.15.2015.01.02.05.38.56 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 02 Jan 2015 05:38:57 -0800 (PST) Message-ID: <54A69F72.6060405@gmail.com> Date: Fri, 02 Jan 2015 14:38:58 +0100 From: Sascha User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: CARP Problem/Bug? on 10.1-RELEASE Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jan 2015 13:39:01 -0000 Hi and a happy new year to everyone! I have problems with my carp setup between two Routers/Firewalls after upgrade from 10.0 RELEASE to 10.1 RELEASE. Before the upgrade my setup worked without any problems! After the upgrade I checked the carp status. Machine 2 (Backup machine) is in Master state for all Interfaces. Machine 1 remains in Backup state. Machine 1 is my primary Machine and should be the master! I restarted both machines several times and checked the rc.conf for errors. But Machine 2 gets every time the master. Machine 1 gets only the master when I turn off Machine 2. All interfaces remain in Master state even when I manually take a Interface for example igb4 with ifconfig on Machine 2 down. Then igb4 on Machine 1 goes into Master state.But the other interfaces remain in Backup state on Machine 1. preemt Option is set to 1 The machines are connected with Link Aggregation to the switch and they do VLAN Tagging. The CARP Interface is the Gateway for the clients. The address on the physical interface is used to bind demons for network services. PF Sync is done over a cross cable direct attached to both machines. PF is not blocking any carp traffic. I Think the problem has something to do with the net.inet.carp.demotion You can see in my output that machine 1 had this value net.inet.carp.demotion: 3840 Is this a bug or does someone see a configuration error in the config? Detailed config(Some output is cleared or omitted): rc.conf Machine 1: ################################################################################################################################################################################ ifconfig_igb0="up" ifconfig_igb1="up" ifconfig_igb2="up" ifconfig_igb3="up" ifconfig_igb4="up" ifconfig_igb5="up" ifconfig_igb4="inet xxx.xxx.0.253 netmask 255.255.255.0" ifconfig_igb4_alias0="vhid 1 pass secret xxx.xxx.0.1/32" # PF Sync Interface ifconfig_igb5="inet xxx.xxx.255.253/30" # Create Virutal Interfaces cloned_interfaces="lagg0 vlan10 vlan25 vlan35 vlan90 vlan95 vlan97 vlan98 vlan99 vlan100 vlan101 vlan102 vlan103 vlan106 vlan107 vlan108 vlan109" # lagg Interface ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 laggport igb3" # VLAN Interfaces ifconfig_vlanxx0="inet xxx.xxxx.xxx.197/27" ifconfig_vlanxx0alias0="inet vhid 2 pass secret xxx.xxx.xxx.199/32 vlan xx0 vlandev lagg0" ifconfig_vlanxx5="inet xxx.xxx.25.253/24" ifconfig_vlanxx5_alias0="inet vhid 3 pass secret xxx.xxx.25.1/32 vlan xx5 vlandev lagg0" ifconfig_vlanxx5="inet xxx.xxx.35.253/24" ifconfig_vlanxx5_alias0="inet vhid 4 pass secret xxx.35.1/32 vlan xx5 vlandev lagg0" ifconfig_vlanxx0="inet xxx.xxx.7.253/21" ifconfig_vlanxx0_alias0="inet vhid 5 pass secret xxx.xxx.7.1/32 vlan xx0 vlandev lagg0" ifconfig_vlanxx5="inet xxx.xxx.95.253/24" ifconfig_vlanxx5_alias0="inet vhid 6 pass secret xxx.xxx.95.1/32 vlan xx5 vlandev lagg0" ifconfig_vlanxx7="inet xxx.xxx.90.253/16" ifconfig_vlanxx7_alias0="inet vhid 7 pass secret xxx.xxx.90.6/32 vlan xx7 vlandev lagg0" ifconfig_vlanxx8="inet xxx.xxx.239.253/21" ifconfig_vlanxx8_alias0="inet vhid 8 pass secret xxx.xxx.232.1/32 vlan xx8 vlandev lagg0" ifconfig_vlanxx9="inet xxx.xxx.0.29/27" ifconfig_vlanxx9_alias0="inet vhid 9 pass secret xxx.xxx.0.1/32 vlan xx9 vlandev lagg0" ifconfig_vlanxx0="inet xxx.xxx.100.253/24" ifconfig_vlanxx0_alias0="inet vhid 10 pass secret xxx.xxx.100.1/32 vlan xx0 vlandev lagg0" ifconfig_vlanxx1="inet xxx.xxx.1.253/24" ifconfig_vlanxx1_alias0="inet vhid 11 pass secret xxx.xxx.1.1/32 vlan xx1 vlandev lagg0" ifconfig_vlanxx2="inet xxx.xxx.116.253/24" ifconfig_vlanxx2_alias0="inet vhid 12 pass secret xxx.xxx.116.1/32 vlan xx2 vlandev lagg0" ifconfig_vlanxx3="inet xxx.xxx.10.253/24" ifconfig_vlanxx3_alias0="inet vhid 13 pass secret xxx.xxx.10.1/32 vlan xx3 vlandev lagg0" ifconfig_vlanxx6="inet xxx.xxx.255.252/16" ifconfig_vlanxx6_alias0="inet vhid 14 pass secret xxx.xxx.255.254/32 vlan xx6 vlandev lagg0" ifconfig_vlanxx7="inet xxx.xxx.90.253/16" ifconfig_vlanxx7_alias0="inet vhid 15 pass secret xxx.xxx.90.140/32 vlan xx7 vlandev lagg0" ifconfig_vlanxx8="inet xxx.xxx.89.12/29" ifconfig_vlanxx8_alias0="inet vhid 16 pass secret xxx.xxx.89.14/32 vlan xx8 vlandev lagg0" ifconfig_vlanxx9="inet xxx.xxx.110.253/24" ifconfig_vlanxx9_alias0="inet vhid 17 pass secret xxx.xxx.110.1/32 vlan xx9 vlandev lagg0" igb4: flags=8943 metric 0 mtu 1500 options=403bb ether 00:1b:21:96:65:78 inet xxx.xxx.0.253 netmask 0xffffff00 broadcast xxx.xxx.0.255 inet xxx.xxx.0.1 netmask 0xffffffff broadcast xxx.xxxx.0.1 vhid 1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active carp: BACKUP vhid 1 advbase 1 advskew 0 vlanxx0: flags=8943 metric 0 mtu 1500 options=303 ether 00:1b:21:96:c0:38 inet xxx.xxx.xxx.197 netmask 0xffffffe0 broadcast xxx.xxx.xxx.223 inet xxx.xxx.xxx.199 netmask 0xffffffff broadcast xxx.xxx.xxx.199 vhid 2 nd6 options=29 media: Ethernet autoselect status: active vlan: 10 parent interface: lagg0 carp: BACKUP vhid 2 advbase 1 advskew 0 root@xxx1:/# sysctl net.inet.carp net.inet.carp.allow: 1 net.inet.carp.preempt: 1 net.inet.carp.log: 2 net.inet.carp.demotion: 3840 net.inet.carp.senderr_demotion_factor: 240 net.inet.carp.ifdown_demotion_factor: 240 Log File when restarting machine 2 root@xxx1:/# less /var/log/messages Jan 2 12:29:11 xxx1 kernel: carp: VHID 15@vlanxx7: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 16@vlanxx8: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 17@vlanxx9: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 1@igb4: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 2@vlanxx0: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 3@vlanxx5: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 4@vlanxx5: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 5@vlanxx0: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 6@vlanxx5: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 7@vlanxx7: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 8@vlanxx8: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 10@vlanxx0: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 9@vlanxx9: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 11@vlanxx1: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 12@vlanxx2: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 13@vlanxx3: BACKUP -> MASTER (master down) Jan 2 12:29:11 xxx1 kernel: carp: VHID 14@vlanxx6: BACKUP -> MASTER (master down) Jan 2 12:29:12 xxx1 kernel: igb5: link state changed to DOWN Jan 2 12:29:14 xxx1 kernel: igb5: link state changed to UP Jan 2 12:29:14 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5' Jan 2 12:29:20 xxx1 kernel: igb5: link state changed to DOWN Jan 2 12:29:22 xxx1 kernel: igb5: link state changed to UP Jan 2 12:29:22 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5' Jan 2 12:29:23 xxx1 kernel: igb5: link state changed to DOWN Jan 2 12:29:24 xxx1 kernel: igb5: link state changed to UP Jan 2 12:29:24 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5' Jan 2 12:29:35 xxx1 kernel: igb5: link state changed to DOWN Jan 2 12:29:37 xxx1 kernel: igb5: link state changed to UP Jan 2 12:29:37 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5' Jan 2 12:31:06 xxx1 kernel: igb5: link state changed to DOWN Jan 2 12:31:08 xxx1 kernel: igb5: link state changed to UP Jan 2 12:31:08 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5' Jan 2 12:31:17 xxx1 kernel: igb5: link state changed to DOWN Jan 2 12:31:19 xxx1 kernel: igb5: link state changed to UP Jan 2 12:31:19 xxx1 devd: Executing '/etc/rc.d/dhclient quietstart igb5' Jan 2 12:31:21 xxx1 kernel: carp: VHID 1@igb4: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 2@vlanxx0: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 3@vlanxx5: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 5@vlanxx0: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 6@vlanxx5: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 9@vlanxx9: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 8@vlanxx8: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 7@vlanxx7: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 14@vlanxx6: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 10@vlanxx0: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 16@vlanxx8: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 11@vlanxx1: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 12@vlanxx2: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 15@vlanxx7: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 13@vlanxx3: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 17@vlanxx9: MASTER -> BACKUP (more frequent advertisement received) Jan 2 12:31:52 xxx1 kernel: carp: VHID 4@vlanxx5: MASTER -> BACKUP (more frequent advertisement received) TCP Dump on machine 1 during restarting machine 2 root@xxx1:/# sudo tcpdump -npi igb4 -T carp dst 224.0.0.18 13:47:55.455580 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167172 13:47:56.881428 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167173 13:47:59.883366 IP xxx.xxx.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167174 13:48:00.894534 IP xxx.xxx.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167175 . . 13:50:07.842383 IP xxx.xxx.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167240 13:50:09.782055 IP xxx.xxx.0.253 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167241 13:50:09.782914 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167242 13:50:11.206767 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167243 13:50:11.510203 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=240 authlen=7 counter=50029484353167244 13:50:12.450678 IP xxx.xxx.0.254 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=100 authlen=7 counter=50029484353167245 ################################################################################################################################################################################ rc.conf Machine 2: ################################################################################################################################################################################ ifconfig_igb0="up" ifconfig_igb1="up" ifconfig_igb2="up" ifconfig_igb3="up" ifconfig_igb4="up" ifconfig_igb5="up" ifconfig_igb4="inet xxx.xxx.0.254 netmask 255.255.255.0" ifconfig_igb4_alias0="vhid 1 advskew 100 pass secret xxx.xxx.0.1/32" # PF Sync Interface ifconfig_igb5="inet 192.168.255.254/30" # Create Virutal Interfaces cloned_interfaces="lagg0 vlan10 vlan25 vlan35 vlan90 vlan95 vlan97 vlan98 vlan99 vlan100 vlan101 vlan102 vlan103 vlan106 vlan107 vlan108 vlan109" # lagg Interface ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 laggport igb3" # VLAN Interfaces ifconfig_vlanxx0="inet xxx.xxx.22.198/27" ifconfig_vlanxx0_alias0="inet vhid 2 advskew 100 pass secret xxx.xxx.22.199/32 vlan 10 vlandev lagg0" ifconfig_vlanxx5="inet xxx.xxx.25.254/24" ifconfig_vlanxx5_alias0="inet vhid 3 advskew 100 pass secret xxx.xxx.25.1/32 vlan 25 vlandev lagg0" ifconfig_vlanxx5="inet xxx.xxx.35.254/24" ifconfig_vlanxx5_alias0="inet vhid 4 advskew 100 pass secret xxx.xxx.35.1/32 vlan 35 vlandev lagg0" ifconfig_vlanxx0="inet xxx.xxx.7.254/21" ifconfig_vlanxx0_alias0="inet vhid 5 advskew 100 pass secret xxx.xxx.0.1/32 vlan 90 vlandev lagg0" ifconfig_vlanxx5="inet xxx.xxx.95.254/24" ifconfig_vlanxx5_alias0="inet vhid 6 advskew 100 pass secret xxx.xxx.95.1/32 vlan 95 vlandev lagg0" ifconfig_vlanxx7="inet xxx.xxx.90.254/16" ifconfig_vlanxx7_alias0="inet vhid 7 advskew 100 pass secret xxx.xxx.90.6/32 vlan 97 vlandev lagg0" ifconfig_vlanxx8="inet xxx.xxx.239.254/21" ifconfig_vlanxx8_alias0="inet vhid 8 advskew 100 pass secret xxx.xxx.232.1/32 vlan 98 vlandev lagg0" ifconfig_vlanxx9="inet xxx.xxx.0.30/27" ifconfig_vlanxx9_alias0="inet vhid 9 advskew 100 pass secret xxx.xxx.0.1/32 vlan 99 vlandev lagg0" ifconfig_vlanxx0="inet xxx.xxx.100.254/24" ifconfig_vlanxx0_alias0="inet vhid 10 advskew 100 pass secret xxx.xxx.100.1/32 vlan 100 vlandev lagg0" ifconfig_vlanxx1="inet xxx.xxx.1.254/24" ifconfig_vlanxx1_alias0="inet vhid 11 advskew 100 pass secret xxx.xxx.1.1/32 vlan 101 vlandev lagg0" ifconfig_vlanxx2="inet xxx.xxx.116.254/24" ifconfig_vlanxx2_alias0="inet vhid 12 advskew 100 pass secret xxx.xxx.116.1/32 vlan 102 vlandev lagg0" ifconfig_vlanxx3="inet xxx.xxx.10.254/24" ifconfig_vlanxx3_alias0="inet vhid 13 advskew 100 pass secret xxx.xxx.10.1/32 vlan 103 vlandev lagg0" ifconfig_vlanxx6="inet xxx.xxx.255.253/16" ifconfig_vlanxx6_alias0="inet vhid 14 advskew 100 pass secret xxx.xxx.255.254/32 vlan 106 vlandev lagg0" ifconfig_vlanxx7="inet xxx.xxx.90.254/16" ifconfig_vlanxx7_alias0="inet vhid 15 advskew 100 pass secret xxx.xxx.90.140/32 vlan 107 vlandev lagg0" ifconfig_vlanxx8="inet xxx.xxx.89.13/29" ifconfig_vlanxx8_alias0="inet vhid 16 advskew 100 pass secret xxx.xxx.89.14/32 vlan 108 vlandev lagg0" ifconfig_vlanxx9="inet xxx.xxx.110.254/24" ifconfig_vlanxx9_alias0="inet vhid 17 advskew 100 pass secret xxx.xxx.110.1/32 vlan 109 vlandev lagg0" igb4: flags=8943 metric 0 mtu 1500 options=403bb ether 00:1b:21:96:66:70 inet xxx.xxx.0.254 netmask 0xffffff00 broadcast xxx.xxx.0.255 inet xxx.xxx.0.1 netmask 0xffffffff broadcast xxx.xxx.0.1 vhid 1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active carp: MASTER vhid 1 advbase 1 advskew 100 vlanxx0: flags=8943 metric 0 mtu 1500 options=303 ether 00:1b:21:96:65:68 inet xxx.xxx.xxx.198 netmask 0xffffffe0 broadcast xxx.xxx.xxx.223 inet xxx.xxx.xxx.199 netmask 0xffffffff broadcast xxx.xxx.xxx.199 vhid 2 nd6 options=29 media: Ethernet autoselect status: active vlan: 10 parent interface: lagg0 carp: MASTER vhid 2 advbase 1 advskew 100 root@xxx2:/# sysctl net.inet.carp net.inet.carp.allow: 1 net.inet.carp.preempt: 1 net.inet.carp.log: 2 net.inet.carp.demotion: 720 net.inet.carp.senderr_demotion_factor: 240 net.inet.carp.ifdown_demotion_factor: 240 root@xxx2:/# ################################################################################################################################################################################