Date: Thu, 9 Jun 2016 19:54:35 +0200 From: Niklaas Baudet von Gersdorff <stdin@niklaas.eu> To: freebsd-net@freebsd.org, "freebsd-questions@freebsd.org" <freebsd-questions@freebsd.org> Subject: Re: Getting CARP to broadcast on a different interface Message-ID: <20160609175434.GA2155@box-hlm-03.niklaas.eu> In-Reply-To: <20160608192347.GE8540@box-hlm-03.niklaas.eu> <20160608163033.GC8540@box-hlm-03.niklaas.eu>
next in thread | previous in thread | raw e-mail | index | archive | help
--dDRMvlgZJXvWKvBx
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
I can't believe it but I managed to do this:
Niklaas Baudet von Gersdorff [2016-06-08 18:30 +0200] :
> Then, I could use devd to assign the public failover IP (that I actually
> wanted to share with CARP on vtnet0) to the public interface vtnet0.
> CARP(4) provides an example on how to use carp status change events for
> additional scripting:
>=20
> --------------------8<--------------------
>=20
> Processing of carp status change events can be set up by using the fol-
> lowing devd.conf rule:
>=20
> notify 0 {
> match "system" "CARP";
> match "subsystem" "[0-9]+@[0-9a-z]+";
> match "type" "(MASTER|BACKUP)";
> action "/root/carpcontrol.sh $subsystem $type";
> };
>=20
> -------------------->8--------------------
>=20
> Depending von $type, carpcontrol.sh could either
>=20
> ifconfig vtnet0 <public-ip> alias
>=20
> or
>=20
> ifconfig vtnet0 <public-ip> -alias
>=20
> I am not sure whether that works but I'll post my solution if I find
> one.
Obviously, the following set-up is very limited to a prober
configuration with CARP on a public network. However, it seems as if
supporting CARP in virtualised environments isn't possible because the
multicast packages would collide with those of other customers in the
same network. So the only solution is to use CARP within a private
network for some pseudo-CARP IP. Then, one can use the devd events that
are triggered if connection via the private network is lost, to assign
and remove the floating public IP that is used for failover.
So, let's see how this can be done. I have three hosts A, B, and C.
I could not manage to get CARP advertising over a virtual private
network of my provider that connects these hosts (CARP used the wrong
MAC addresses so the packets weren't translated correctly). But what
I managed to get going is CARP advertising over my tinc VPN tap0.
So if any host comes online and connects to the other hosts on the VPN,
tinc will assign a CARP vhid on tap0. This is
/usr/local/etc/tinc/host-up on host A:
--------------------8<--------------------
ifconfig $INTERFACE vhid 1 advbase 01 advskew 000 pass <some-pasword> 10.99=
=2E99.99 alias
-------------------->8--------------------
On host B and C the file looks exactly the same except that advbase is
five seconds longer each (06 and 11 respectively). (While testing the
set-up I realised that latency is too high, so that advskew wouldn't
suffice.)
I use exactly the same devd.conf as quoted above. I put it under
/usr/local/etc/devd.
Further, I did the following:
sysctl net.inet.carp.preempt=3D1
which should be added to /etc/sysctl.conf.
/root/carpcontrol.sh is on every host and looks like the following:
--------------------8<--------------------
1 #!/bin/sh
2 =20
3 subsystem=3D$1
4 type=3D$2
5 vhid=3D"1"
6 c_if=3D"tap0" # CARP interface
7 =20
8 e_if=3D"vtnet0" # external interface
9 IPv4=3D"<floating-IPv4>"
10 IPv4_net=3D"/24"
11 IPv6=3D"<floating-IPv6>"
12 IPv6_net=3D"/64"
13 =20
14 log_tag=3D"carpcontrol.sh"
15 =20
16 remove_floating_ip() {
17 if ifconfig $e_if | grep $IPv4 >/dev/null
18 then
19 ifconfig $e_if $IPv4$IPv4_net -alias && logger -t $log_tag =
"Removed $IPv4 on $e_if."
20 fi
21 if ifconfig $e_if | grep $IPv6 >/dev/null
22 then
23 ifconfig $e_if inet6 $IPv6$IPv6_net -alias && logger -t $lo=
g_tag "Removed $IPv6 on $e_if."
24 fi
25 }
26 =20
27 add_floating_ip() {
28 ifconfig $e_if $IPv4$IPv4_net alias && logger -t $log_tag "Assi=
gned $IPv4 on $e_if."
29 ifconfig $e_if inet6 $IPv6$IPv6_net alias && logger -t $log_tag=
"Assigned $IPv6 on $e_if."
30 }
31 =20
32 if [ "$subsystem" =3D "$vhid@$c_if" ]
33 then
34 case $type in=20
35 "MASTER")
36 add_floating_ip
37 ;;
38 "BACKUP")
39 remove_floating_ip
40 ;;
41 esac
42 elif [ "$subsystem" =3D "tinc-down" ]
43 then
44 remove_floating_ip
45 fi
-------------------->8--------------------
Probably that's not the best way to do it and probably it could be
extended but my sh-skills are quite bad. This works though.
The script should be self-explanatory; the only ting that might be
unclear are lines 42-45. This is triggered by /usr/local/etc/tinc-down:
--------------------8<--------------------
/root/carpcontrol.sh tinc-down
-------------------->8--------------------
So, in case I stop tincd on purpose, the floating IP will be removed and
becomes free for the other machines. The second host will take over the
IP after some seconds.
Obviously, there are quite some drawbacks to this but it serves my
purpose. Since the machines are virtualised I don't expect them to down
by some hardware failure. But I do need failover in case I want to
upgrade one machines and need to reboot it.
In case I need to do some maintenance, I cut the machine in question
=66rom the private network. For some seconds services won't be reachable
but after a short while host B will take over. I can upgrade host A and
restart it eventually. When host A runs again, it will connect to the
tinc VPN and while advertise CARP on that network. So, host A will
becomes MASTER again, host B will switch to BACKUP, and C will remain
BACKUP.
Nonetheless, this is not a perfect solution. "High-availability" (these
quotes are on purpose) as achieved this way, depends on the connectivity
between the tinc nodes. E.g., if tinc malfunctions and doesn't shutdown
properly, the floating IP will stay assigned to the public interface,
the other client will loose connectivity to the node, and an additional
MASTER will rise. In addition, I had problems using a faster advbase
because of latency, I guess, thus there will be some downtime if any
MASTER goes down. In this case, the problem was that two hosts could not
decide who should be MASTER because there were some connectivity
problems between them on the VPN.
The crux is that CARP, as implemented in FreeBSD, cannot advertise on
unicast. If it could, I would have been able to assign it to the private
virtual interface that connects the virtual machines. However, CARP, as
implemented in OpenBSD, can do so using a feature called "carppeer".
There have been requests whether/when that will be brought to FreeBSD
[1-3] but the questions remained without responses.=20
1: https://lists.freebsd.org/pipermail/freebsd-net/2013-November/037106.html
2: https://lists.freebsd.org/pipermail/freebsd-net/2011-February/027999.html
3: https://lists.freebsd.org/pipermail/freebsd-pf/2009-December/005486.html
So, where can I post feature requests? :-) Without a way for unicast
CARP packets, FreeBSD is of much less value for virtualised environments
where multicast packets are an issue because they are blocked by
providers. So, setting up a failover set-up becomes a mess (such as the
one above).
Niklaas Baudet von Gersdorff [2016-06-08 21:23 +0200] :
> For that to work I must bind processes to non-local IP addresses. How do
> I do that?
>=20
> I found this
>=20
> https://lists.freebsd.org/pipermail/freebsd-hackers/2011-January/034033.h=
tml
>=20
> with some recommendations to do so with ipfw. Can I do something similar
> with pf? Or is there even another solution for binding to non-local
> addresses?
No longer necessary. I use a pf rule that redirect any traffic arriving
on a specific port (this could also be any traffic arriving on the
pseudo-CARP IP) to the jail where the load-balancer is listening.
Anyway, I'm happy for any comments and further suggestions.
Niklaas
--dDRMvlgZJXvWKvBx
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJXWa1UAAoJEG2fODeJrIU/JcQP/if7WQQrKliePAZBWeTLOdww
PDEfG8NzF/ValyiMl8ihSKIZCnWZulMaWWbHrzE0eQzDVunLkG4ZbtmF4eyXPyC8
hpSKUFnlbsBmogSyLegxjP3U5XxQR4y8HJKXLvhNlYfjWta9q3Vuvlg/SL7Lzp3J
1hvwl/jsuexM/qwRaVT4ueRq3UM1lvtS2oun605JAIkw3bhQOoCVVCYX9VQGNriu
vzPIVAp9/NP6u44mJRnyxU0ChJDdXQF8evggkACIaRmZV0Jgg3oKuf71jQ0k7d0X
5F9ODt7ZQLrLonQ0jd9eMn9bEMmYStQgBhQywuTCFneoe7ahuM/ds9/hpYpx3TRR
jiiuy1hdTvgL+lYV+N3cVmpka+k9c1od2VBGKmf8nzwKadsrv9Yi9ywUYgwjaKnB
FdwqEBq/fEd5PiiljmQDsELfacFP2cRbkN3W0dvSEwUVmQlXNIh0abcoOCSzag6Z
tnxc8ebTKHcP6QUMQX12l9V7ehU5J2KVCG3ABs2VQGcmrXJBPdx+Vm37oIqf/RCW
LS0wZ3zZxGV06NylL5fYgo/8gi7XIiW9MxpkdNlyRLVZPB0ji0eP1NzXyaK5jY+F
YXwa92mMMZ35biLQGxXr1UwigkFKnphsqCWuXHhML4wJ/e+cF7XZbDMvcnx5Z9Kp
/+1P6QX1apt6qlGHhmFC
=nede
-----END PGP SIGNATURE-----
--dDRMvlgZJXvWKvBx--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160609175434.GA2155>