Date: Sat, 1 Oct 2011 18:18:34 +0200 From: Damien Fleuriot <ml@my.gd> To: johan Hendriks <joh.hendriks@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: CARP interfaces and mastership issue Message-ID: <CAE63ME7_3R3bywi8HfB1A5k4ckhvf6d48j_-0UE1ZGifFdKx5g@mail.gmail.com> In-Reply-To: <4E86D787.9050207@gmail.com> References: <4E71C059.5060404@hi-media.com> <4E84627B.2050609@my.gd> <CAE63ME71mXfHeV7hxvtkv8q-m0rVQ8-z7aNfF61Cb0poVJCpRQ@mail.gmail.com> <4E848592.5050300@my.gd> <4E86D787.9050207@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1 October 2011 11:04, johan Hendriks <joh.hendriks@gmail.com> wrote:
> Op 29-09-11 16:49, Damien Fleuriot schreef:
>>>
>>>
>>> Quick follow-up again.
>>>
>>> This is the code for sys/netinet/ip_carp.c on FreeBSD 8.2, OpenBSD
>>> 3.8, OpenBSD 3.9 in function carp_setrun(struct carp_softc *sc,
>>> sa_family_t af)
>>>
>>>
>>>
>>> FREEBSD 8.2-PRERELEASE with init + preempt =3D> =A0auto MASTER bug
>>> Function starts at line 1371.
>>> ---
>>> =A0 =A0 =A0 =A0 switch (sc->sc_state) {
>>> =A0 =A0 =A0 =A0 case INIT:
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (carp_opts[CARPCTL_PREEMPT]&& =A0!ca=
rp_suppress_preempt)
>>> {
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_send_ad_locked(sc)=
;
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_send_arp(sc);
>>> #ifdef INET6
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_send_na(sc);
>>> #endif /* INET6 */
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 CARP_LOG("%s: INIT -> =
=A0MASTER (preempting)\n",
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 SC2IFP(sc)->if_=
xname);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_set_state(sc, MAST=
ER);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_setroute(sc, RTM_A=
DD);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } else {
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 CARP_LOG("%s: INIT -> =
=A0BACKUP\n",
>>> SC2IFP(sc)->if_xname);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_set_state(sc, BACK=
UP);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_setroute(sc, RTM_D=
ELETE);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_setrun(sc, 0);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break;
>>> ---
>>>
>>> OPENBSD 3.8 with init + preempt =3D> =A0auto MASTER bug
>>> Function starts at line 1293.
>>> ---
>>> =A0 =A0 =A0 =A0 case INIT:
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (carp_opts[CARPCTL_PREEMPT]&& =A0!ca=
rp_suppress_preempt)
>>> {
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_set_state(sc, MAST=
ER);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_setroute(sc, RTM_A=
DD);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_send_ad(sc);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_send_arp(sc);
>>> #ifdef INET6
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_send_na(sc);
>>> #endif /* INET6 */
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } else {
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_set_state(sc, BACK=
UP);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_setroute(sc, RTM_D=
ELETE);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_setrun(sc, 0);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break;
>>> ---
>>>
>>>
>>>
>>> OPENBSD 3.9 with bug fixed
>>> Function starts at line 1348.
>>> ---
>>> =A0 =A0 =A0 =A0 switch (sc->sc_state) {
>>> =A0 =A0 =A0 =A0 case INIT:
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_set_state(sc, BACKUP);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_setroute(sc, RTM_DELETE);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 carp_setrun(sc, 0);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break;
>>> ---
>>>
>>>
>>> It looks like the root cause is there.
>>>
>>> I'll rebuild and test, keep you updated.
>>
>>
>>
>> Find below my test results with the OpenBSD39 implementation which
>> forces an INIT -> =A0BACKUP transition regardless of preempt.
>>
>>
>>
>>
>>
>>
>> # sysctl net.inet.carp.preempt
>> net.inet.carp.preempt: 1
>>
>> # sysctl net.inet.carp.suppress_preempt
>> net.inet.carp.suppress_preempt: 0
>>
>> # ifconfig carp17
>> carp17: flags=3D8<LOOPBACK> =A0metric 0 mtu 1500
>> =A0 =A0 =A0 =A0inet 46.182.[snip] netmask 0xffffffff
>> =A0 =A0 =A0 =A0inet 46.182.[snip] netmask 0xffffffff
>> =A0 =A0 =A0 =A0inet 46.182.[snip] netmask 0xffffffff
>> =A0 =A0 =A0 =A0inet 46.182.[snip] netmask 0xffffffff
>> =A0 =A0 =A0 =A0inet 46.182.[snip] netmask 0xffffffff
>> =A0 =A0 =A0 =A0carp: INIT vhid 117 advbase 1 advskew 200
>>
>> # ifconfig carp17 up; ./check_carp17_status.sh
>> count: 0 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 1 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 2 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 3 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 4 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 5 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 6 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 7 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 8 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 9 =A0 =A0 =A0 =A0carp: BACKUP vhid 117 advbase 1 advskew 200
>> count: 10 =A0 =A0 =A0 carp: BACKUP vhid 117 advbase 1 advskew 200
>>
>> # dmesg
>> carp17: INIT -> =A0BACKUP
>> carp17: link state changed to DOWN
>>
>>
>>
>> Looks like it works.
>>
>> I'm afraid I cannot test actual preemption by shutting down a physical
>> interface on the MASTER, because they're actually used in production and
>> I've got users logged in to their VPN on pf1.
>>
>> I see no reason this should break anything however, it only forces the
>> CARP interface to assume a BACKUP state right after INIT, as it normally
>> should, regardless of preemption.
>>
>> I'm filling a PR + patch.
>> _______________________________________________
>> freebsd-stable@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org=
"
>
> Do you have the PR number for me and the patch, i have the same problem a=
nd
> like it to be fixed.
>
> regards
> Johan Hendriks
>
>
http://www.freebsd.org/cgi/query-pr.cgi?pr=3D161123
Here you go.
Please note I do not have extensive C programming experience.
I have simply removed the part that makes the CARP interface
immediately transition from INIT to MASTER when preemption is enabled,
and replaced it with an INIT->BACKUP transition like in OpenBSD3.9 in
which the bug is fixed.
I cannot guarantee the absence of side effects, although I couldn't
find any in my testing.
Also note that if you update your sources the patch will obviously be
gone, take care when upgrading.
You'll want to rebuild the kernel at the very least, to be on the safe
side I also did the world, you never know.
Regards
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAE63ME7_3R3bywi8HfB1A5k4ckhvf6d48j_-0UE1ZGifFdKx5g>