Date: Thu, 02 Jun 2011 21:10:04 -0400 From: Steve Polyack <korvus@comcast.net> To: jwd@SlowBlink.Com Cc: freebsd-net@freebsd.org Subject: Re: Production use of carp? Message-ID: <4DE8346C.6000306@comcast.net> In-Reply-To: <20110603001424.GA81480@slowblink.com> References: <20110602203940.GA80549@slowblink.com> <20110603003719.61292e9e@davenulle.org> <20110603001424.GA81480@slowblink.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 6/2/2011 8:14 PM, John De Boskey wrote: > ----- Patrick Lamaiziere's Original Message ----- >> Le Thu, 2 Jun 2011 16:39:40 -0400, >> John<jwd@SlowBlink.Com> a ?crit : >> >>> Instead of running carp on the external interfaces as below: >>> >>> ifconfig_cxgb0="inet 10.24.99.11 netmask 255.255.0.0" # System 1 >>> physical ip ifconfig_cxgb0="inet 10.24.99.12 netmask 255.255.0.0" # >>> System 2 physcial ip ifconfig_carp1="vhid 1 pass zfscarp1 advbase 1 >>> advskew 100 10.24.99.13 netmask 255.255.0.0" # HA ip used by clients >>> ... we instead connect a direct cross-over cable between the two >>> systems providing HA/Failover and use a private (backside) network: >> I've missed this... >> >> As the purpose of carp is to provide a shared ip on a network, I don't >> see why you are trying to use it on a cross-over network, between >> only two machines? It seems useless to me. >> >> Regards. > I have separate scripts which monitor the external interfaces on > the two systems. If, for instance, one of the public ip addresses, > 10.24.99.11 were to go down, the monitor script issues a ifconfig > carp1 down, causing the service to shift over to partner system. > These are actually hooked up with devd. These scripts also shift > the virtual ip from one system to the other (as an ifconfig alias). > > I'm trying to avoid the controlling interface being on the external > interfaces. As I said previously, exposing the vhid values to the > external net (company internal net) means we have to be careful > not to allow the same vhid twice, and to avoid the problem of the > switch going down and both systems thinking they should be the > master. Just to clarify, the reason using the same VHID twice on the same network is bad news is because it is what is used to generate the virtual MAC address which the master system will respond to ARP requests with. Unfortunately, the password doesn't come into play here. Why worry about when the switch goes down? Shouldn't that eliminate any traffic which could cause changes to the exported filesystems? What if somebody knocks out your crossover cable and you have both systems thinking they are master AND are accessible (in some sense, since the MAC is going to flap between switchports). > In general, I attach exported filesystem services to different > carp interfaces. I load balance them between the two HA servers. > For instance, /vol/data1 is on carp1, /vol/data2 is on carp2. > Under normal circumstance, /vol/data1 is "owned" by system A, > and /vol/data2 is owned by sysem B. Issuing 'ifconfig carp1 down' > on system A causes the export of /vol/data1 to shift over to > system B at which point maintanence can be done on system A. > The only problem is taking down system A causes the carp interfaces > on system B to go down/up a few times. > > Does that help a bit? I think your problem lies with the crossover cable. When the crossover interface goes down (as in link state), the CARP interface on the system which is still online is going to flap. Even if you do not specify 'carpdev cxgb0', the system will locate the proper physical interface to associate your CARP interface with. If this is really what you want, you may have to hack the kernel to have CARP ignore link state changes on the associated physical interface. I would suggest just using CARP on the interface which connect to the switches. Just as each set of two storage units will likely share a Virtual IP which is unique to them, they should also share a unique VHID. Yes, you only have 1-255, but since the VHID only controls the virtual MAC, reusing a VHID in a different VLAN should not matter (provided you are using a different password or the VLANs are also in different multicast domains). Perhaps you can create a VLAN local to the switch just for this purpose? Setup a vlan(4) interface on each system just for this purpose and attach the CARP device to it. Overall, I think you're trying to use CARP to accomplish something other than what it was designed for. Why not just run a ping on the crossover interface every second and use that to trigger your switchovers? You could get away with a pretty low timeout. Add in some kind of way to force a maintenance-mode on a box and I think you'd have a solution that does what you want without the pitfalls you are seeing. - Steve
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4DE8346C.6000306>