From owner-freebsd-cluster@FreeBSD.ORG Tue Mar 27 20:47:31 2007 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D75C516A402 for ; Tue, 27 Mar 2007 20:47:31 +0000 (UTC) (envelope-from ross@virtualgeek.net) Received: from achilles.virtualgeek.net (perseus.demon.co.uk [83.104.128.109]) by mx1.freebsd.org (Postfix) with ESMTP id 8A2FC13C4D9 for ; Tue, 27 Mar 2007 20:47:31 +0000 (UTC) (envelope-from ross@virtualgeek.net) Received: from virtualgeek.net (achilles.virtualgeek.net [127.0.0.1]) by achilles.virtualgeek.net (Postfix) with ESMTP id 1BDFE104B3E for ; Sun, 25 Mar 2007 19:23:50 +0100 (BST) Received: from 83.104.128.109 (SquirrelMail authenticated user ross.virtualgeek) by virtualgeek.net with HTTP; Sun, 25 Mar 2007 19:23:50 +0100 (BST) Message-ID: <48279.83.104.128.109.1174847030.squirrel@virtualgeek.net> Date: Sun, 25 Mar 2007 19:23:50 +0100 (BST) From: "Ross Draper" To: freebsd-cluster@freebsd.org User-Agent: SquirrelMail/1.4.4 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-15 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Subject: Vrrp/CARP/UCarp Problems X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2007 20:47:31 -0000 Hi guys I was wondering if I could get some advice from those of you who have successfully implemented ip address failover systems such as carp and freevrrpd. I am trying to set up a high availability web loadbalancer using a pair of freebsd 6.2 boxes. I have tried a number of ways to perform failover but always seem to be hitting a problem. UCARP - Pro's:This would be my ideal solution as the startup/shutdown scripts enable me to stop and start my applications and add aliases to adaptors easily. Cons: When the backup box is rebooted it always comes up advertising itself as the master then after a few seconds revers to backup, although I was under the impression it was supposed to wait and listen for advertisements(it doesnt seem to). Its initial gratuitous arp as a master is sufficient to poison any traffic from the local router to the shared ip address. Only solution was to use arp-sk to send gratuitous arps every few secs, however, arp-sk was a bit flakey and it was a bodge. CARP - Pro's: stable and built into the kernel. Could enable acive/active arp load sharing at a later point. Cons: There is a Freebsd bug (I've seen it discussed on the lists where the creation and destroyal of a carp interface causes a kernel panic. Also, there is no support for start/stop scripts. Freevrrpd - Pros: Mac address changing removes some of the arp timeout issues/gratuitus arp problems and it supports start/stop scripts Cons: I'm finding that upon rebooting the backup unit it correctly starts as a backup, then three seconds later syslogs that it is the master and changes its mac address accordingly. although a sniff of the network traffic indicates it is sending the right advertisements, it never goes into backup mode again. So, what am I doing wrong? are these the experiences others have had or are there more suitable options? the loadbalancers are all single homed and I have tried a mixture of xl, bge and fxp cards. Also, any links to a perl based gratuitous arp utils would be great Any help/suggestions much appreciated. Ross From owner-freebsd-cluster@FreeBSD.ORG Wed Mar 28 06:01:06 2007 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 57D5F16A410 for ; Wed, 28 Mar 2007 06:01:06 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from redbull.bpaserver.net (redbullneu.bpaserver.net [213.198.78.217]) by mx1.freebsd.org (Postfix) with ESMTP id 0FC6913C469 for ; Wed, 28 Mar 2007 06:01:06 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from outgoing.leidinger.net (p54a5f07f.dip.t-dialin.net [84.165.240.127]) by redbull.bpaserver.net (Postfix) with ESMTP id 0A9242E215; Wed, 28 Mar 2007 07:31:58 +0200 (CEST) Received: from webmail.leidinger.net (webmail.Leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 3453C5B4817; Wed, 28 Mar 2007 07:31:55 +0200 (CEST) Received: (from www@localhost) by webmail.leidinger.net (8.13.8/8.13.8/Submit) id l2S5VtZc017859; Wed, 28 Mar 2007 07:31:55 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde MIME library) with HTTP; Wed, 28 Mar 2007 07:31:54 +0200 Message-ID: <20070328073154.j62ahkhwggc04wg4@webmail.leidinger.net> X-Priority: 3 (Normal) Date: Wed, 28 Mar 2007 07:31:54 +0200 From: Alexander Leidinger To: Ross Draper References: <48279.83.104.128.109.1174847030.squirrel@virtualgeek.net> In-Reply-To: <48279.83.104.128.109.1174847030.squirrel@virtualgeek.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.1.3) / FreeBSD-7.0 X-BPAnet-MailScanner-Information: Please contact the ISP for more information X-BPAnet-MailScanner: Found to be clean X-BPAnet-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-14.71, required 8, BAYES_00 -15.00, DK_POLICY_SIGNSOME 0.00, FORGED_RCVD_HELO 0.14, TW_RR 0.08, TW_VR 0.08) X-BPAnet-MailScanner-From: alexander@leidinger.net X-Spam-Status: No Cc: freebsd-cluster@freebsd.org Subject: Re: Vrrp/CARP/UCarp Problems X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 06:01:06 -0000 Quoting Ross Draper (from Sun, 25 Mar 2007 19:23:50 +0100 (BST)): > Freevrrpd - Pros: Mac address changing removes some of the arp timeout > issues/gratuitus arp problems and it supports start/stop scripts > Cons: I'm finding that upon rebooting the backup unit it correctly starts > as a backup, then three seconds later syslogs that it is the master and > changes its mac address accordingly. although a sniff of the network > traffic indicates it is sending the right advertisements, it never goes > into backup mode again. That's not normal, can you post your config? Bye, Alexander. -- Economics is extremely useful as a form of employment for economists. -- John Kenneth Galbraith http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-cluster@FreeBSD.ORG Wed Mar 28 21:29:42 2007 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7883B16A403 for ; Wed, 28 Mar 2007 21:29:42 +0000 (UTC) (envelope-from ross@virtualgeek.net) Received: from achilles.virtualgeek.net (perseus.demon.co.uk [83.104.128.109]) by mx1.freebsd.org (Postfix) with ESMTP id E339013C46C for ; Wed, 28 Mar 2007 21:29:41 +0000 (UTC) (envelope-from ross@virtualgeek.net) Received: from 192.168.5.12 (achilles.virtualgeek.net [127.0.0.1]) by achilles.virtualgeek.net (Postfix) with ESMTP id 2EAD0104B41; Wed, 28 Mar 2007 22:29:36 +0100 (BST) Received: from 192.168.5.13 (SquirrelMail authenticated user ross.virtualgeek) by 192.168.5.12 with HTTP; Wed, 28 Mar 2007 22:29:36 +0100 (BST) Message-ID: <2615.192.168.5.13.1175117376.squirrel@192.168.5.12> In-Reply-To: <20070328073154.j62ahkhwggc04wg4@webmail.leidinger.net> References: <48279.83.104.128.109.1174847030.squirrel@virtualgeek.net> <20070328073154.j62ahkhwggc04wg4@webmail.leidinger.net> Date: Wed, 28 Mar 2007 22:29:36 +0100 (BST) From: "Ross Draper" To: "Alexander Leidinger" User-Agent: SquirrelMail/1.4.4 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-15 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: Ross Draper , freebsd-cluster@freebsd.org Subject: Re: Vrrp/CARP/UCarp Problems X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2007 21:29:42 -0000 > Quoting Ross Draper (from Sun, 25 Mar 2007 > 19:23:50 +0100 (BST)): > >> Freevrrpd - Pros: Mac address changing removes some of the arp timeout >> issues/gratuitus arp problems and it supports start/stop scripts >> Cons: I'm finding that upon rebooting the backup unit it correctly >> starts >> as a backup, then three seconds later syslogs that it is the master and >> changes its mac address accordingly. although a sniff of the network >> traffic indicates it is sending the right advertisements, it never goes >> into backup mode again. > > That's not normal, can you post your config? > > Bye, > Alexander. > > -- > Economics is extremely useful as a form of employment for economists. > -- John Kenneth Galbraith > > http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 > http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 > Hi Alexander Thanks for your reply. My config is: Master [VRID] serverid = 5 interface = em0 useVMAC = yes sendgratuitousarp = yes priority = 255 addr=10.73.122.205/24, 10.2.200.1/32 masterscript = /usr/local/etc/master.sh backupscript = /usr/local/etc/backup.sh Backup [VRID] serverid = 5 interface = em0 useVMAC = yes sendgratuitousarp = yes priority = 250 addr=10.73.122.205/24, 10.2.200.1/32 masterscript = /usr/local/etc/master.sh backupscript = /usr/local/etc/backup.sh I posted this message to the freebsd-net group as well(actually by mistake, but thats another story), and as part of my investigations from replies received there, I was starting to believe that my problem was related to different types of network card/drivers. However, I have now built two boxes on identical hardware as well as a mixture of xl/em and things work fine. I am now starting to believe that the problems I experienced in freevrrpd may be as a result of having carp compiled into my kernel. The two new machines I have now built are stock 6.2, with no carp in the kernel. (Note, the original machines had no carp interfaces defined on them when using freevrrpd) I cant confirm fully until I am back in the office, but I will post back as soon as I am able. Kind Regards Ross From owner-freebsd-cluster@FreeBSD.ORG Fri Mar 30 06:44:18 2007 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4EA3716A401 for ; Fri, 30 Mar 2007 06:44:18 +0000 (UTC) (envelope-from dkirhlarov@mail.oilspace.com) Received: from office.oilspace.com (ns2.oilspace.com [194.129.65.230]) by mx1.freebsd.org (Postfix) with ESMTP id 10C7013C483 for ; Fri, 30 Mar 2007 06:44:18 +0000 (UTC) (envelope-from dkirhlarov@mail.oilspace.com) Received: from localhost.mow.oilspace.com (office-lc.mow.oilspace.com [81.222.156.11]) by office.oilspace.com (Postfix) with ESMTP id 8C100136CDC for ; Fri, 30 Mar 2007 06:21:13 +0000 (UTC) Received: from localhost.mow.oilspace.com (localhost.mow.oilspace.com [127.0.0.1]) by localhost.mow.oilspace.com (8.13.8/8.13.8) with ESMTP id l2U6L44u025082 for ; Fri, 30 Mar 2007 10:21:06 +0400 (MSD) (envelope-from dkirhlarov@localhost.mow.oilspace.com) Received: (from dkirhlarov@localhost) by localhost.mow.oilspace.com (8.13.8/8.13.8/Submit) id l2S79ofd006835 for freebsd-cluster@freebsd.org; Wed, 28 Mar 2007 11:09:50 +0400 (MSD) (envelope-from dkirhlarov) Date: Wed, 28 Mar 2007 11:09:50 +0400 From: Dmitriy Kirhlarov To: freebsd-cluster@freebsd.org Message-ID: <20070328070947.GA6576@dkirhlarov.mow.oilspace.com> Mail-Followup-To: freebsd-cluster@freebsd.org References: <48279.83.104.128.109.1174847030.squirrel@virtualgeek.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48279.83.104.128.109.1174847030.squirrel@virtualgeek.net> X-Mailer: Mutt-ng devel (2005-03-13) based on Mutt 1.5.9 X-Operating-System: FreeBSD 6.2-STABLE User-Agent: mutt-ng/devel-r804 (FreeBSD) Subject: Re: Vrrp/CARP/UCarp Problems X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Mar 2007 06:44:18 -0000 On Sun, Mar 25, 2007 at 07:23:50PM +0100, Ross Draper wrote: > CARP - Pro's: stable and built into the kernel. Could enable acive/active > arp load sharing at a later point. > Cons: There is a Freebsd bug (I've seen it discussed on the lists where > the creation and destroyal of a carp interface causes a kernel panic. I use it without panics half year. How many carp interfaces must be configured for reproduce panic? Is net.inet.carp.preempt used? > Also, there is no support for start/stop scripts. Use ports/net/ifstated for run init scripts. > Freevrrpd - Pros: Mac address changing removes some of the arp timeout > issues/gratuitus arp problems and it supports start/stop scripts > Cons: I'm finding that upon rebooting the backup unit it correctly starts > as a backup, then three seconds later syslogs that it is the master and > changes its mac address accordingly. although a sniff of the network > traffic indicates it is sending the right advertisements, it never goes > into backup mode again. Look like you have same "priority" on both nodes. It's work only with CARP. WBR. Dmitriy From owner-freebsd-cluster@FreeBSD.ORG Sat Mar 31 22:27:18 2007 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A7E5516A40F for ; Sat, 31 Mar 2007 22:27:18 +0000 (UTC) (envelope-from bseklecki@collaborativefusion.com) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by mx1.freebsd.org (Postfix) with ESMTP id 7EAFF13C459 for ; Sat, 31 Mar 2007 22:27:18 +0000 (UTC) (envelope-from bseklecki@collaborativefusion.com) Received: from [192.168.4.6] (c-67-171-75-90.hsd1.pa.comcast.net [67.171.75.90]) (AUTH: LOGIN seklecki, TLS: TLSv1/SSLv3,128bits,RC4-MD5) by wingspan with esmtp; Sat, 31 Mar 2007 18:17:06 -0400 id 00056403.460EDDE2.0000FB8C From: "Brian A. Seklecki" To: Ross Draper In-Reply-To: <48279.83.104.128.109.1174847030.squirrel@virtualgeek.net> References: <48279.83.104.128.109.1174847030.squirrel@virtualgeek.net> Organization: Collaborative Fusion, Inc. Date: Sat, 31 Mar 2007 17:17:20 -0500 Message-Id: <1175379440.7585.58.camel@ingress> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 7bit X-Mailer: Evolution 2.2.3-10mdk Cc: freebsd-cluster@freebsd.org Subject: Re: Vrrp/CARP/UCarp Problems X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Mar 2007 22:27:18 -0000 You could put an OpenBSD or FreeBSD box runnig pf(4) in "front" of your web server cluster. You setup your public IP anchor and a service-VIP for your web service application. Then you do a RDR nat into a pf(4) table. You set the contents of the table based on a shell script that checks the health of the system. I suppose you could carp between the two RDR boxes; keep state tables even too. I.e., carp was never designed to move an HA L4 address between two systems. Only to provide a HA L4 IP gateway. Bob Beck did a great presentation on this at NYCBSDcon 06. Google it and grab his slides. ~BAS On Sun, 2007-03-25 at 19:23 +0100, Ross Draper wrote: > Hi guys > > I was wondering if I could get some advice from those of you who have > successfully implemented ip address failover systems such as carp and > freevrrpd. > > I am trying to set up a high availability web loadbalancer using a pair of > freebsd 6.2 boxes. I have tried a number of ways to perform failover but > always seem to be hitting a problem. > > UCARP - Pro's:This would be my ideal solution as the startup/shutdown > scripts enable me to stop and start my applications and add aliases to > adaptors easily. > Cons: When the backup box is rebooted it always comes up advertising > itself as the master then after a few seconds revers to backup, although I > was under the impression it was supposed to wait and listen for > advertisements(it doesnt seem to). Its initial gratuitous arp as a master > is sufficient to poison any traffic from the local router to the shared ip > address. Only solution was to use arp-sk to send gratuitous arps every few > secs, however, arp-sk was a bit flakey and it was a bodge. > > CARP - Pro's: stable and built into the kernel. Could enable acive/active > arp load sharing at a later point. > Cons: There is a Freebsd bug (I've seen it discussed on the lists where > the creation and destroyal of a carp interface causes a kernel panic. > Also, there is no support for start/stop scripts. > > Freevrrpd - Pros: Mac address changing removes some of the arp timeout > issues/gratuitus arp problems and it supports start/stop scripts > Cons: I'm finding that upon rebooting the backup unit it correctly starts > as a backup, then three seconds later syslogs that it is the master and > changes its mac address accordingly. although a sniff of the network > traffic indicates it is sending the right advertisements, it never goes > into backup mode again. > > So, what am I doing wrong? are these the experiences others have had or > are there more suitable options? the loadbalancers are all single homed > and I have tried a mixture of xl, bge and fxp cards. > > Also, any links to a perl based gratuitous arp utils would be great > > Any help/suggestions much appreciated. > > Ross > > > _______________________________________________ > freebsd-cluster@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-cluster > To unsubscribe, send any mail to "freebsd-cluster-unsubscribe@freebsd.org"