From owner-freebsd-cluster@FreeBSD.ORG Fri Feb 18 21:01:40 2011 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0BF1C10656CA for ; Fri, 18 Feb 2011 21:01:40 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id BDC928FC1F for ; Fri, 18 Feb 2011 21:01:39 +0000 (UTC) Received: by gyh4 with SMTP id 4so306632gyh.13 for ; Fri, 18 Feb 2011 13:01:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=+5CifwGUIQNJc2MhxaTa2Gfhzz+SY4ODgw/7Oefkqsg=; b=X9yK/B6VeKCHbx4TsLzY8tMF4JBQJhwdCnQf3hwex+ftvIN/0vKbUfMlFxlKqaTlfH FIsGjilcQtCGi9kY8AprBgL2fg8YyA5s7ZlkmKNxNfWxuRD3WCochg18CnHMxDwKy4wp H/l36suipWhNAO+h+DBRzH0QC3gVDlWXE+VCw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=RFZjwjP4WJZXZ/Zue66DtLzBR5gmhkXfFEE0a4V3Grid68P/Xguv0f8gkd/uFt9Vl4 X7VzZNPdE3alAVdImhX+GjYbUgY+VbZTdmEx8i1E4B8pS0aFE/sfhPG3t0p7L7VL/gAl am1bt9f/trAEkvzMb5tUJqNpqczU4YI8Ojyoo= MIME-Version: 1.0 Received: by 10.91.209.6 with SMTP id l6mr1681741agq.42.1298060999983; Fri, 18 Feb 2011 12:29:59 -0800 (PST) Received: by 10.90.32.20 with HTTP; Fri, 18 Feb 2011 12:29:59 -0800 (PST) In-Reply-To: <1298020090.18890.1684.camel@pcdenny> References: <1298020090.18890.1684.camel@pcdenny> Date: Fri, 18 Feb 2011 12:29:59 -0800 Message-ID: From: Freddie Cash To: Denny Schierz Content-Type: text/plain; charset=UTF-8 Cc: freebsd-cluster@freebsd.org Subject: Re: Build failover ZFS, like HA-Storage from Solaris X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Feb 2011 21:01:40 -0000 On Fri, Feb 18, 2011 at 1:08 AM, Denny Schierz wrote: > we're searching for an alternative failover solution with ZFS. We have > two nodes connected to _one_ SAS Storage (so, no DRBD or anything > possible) and we want, to export zfs volumes via ISCSI to other systems. > > If the primary node fails, the ZFS Pool (which also does 8 times raidz2) > has to be move to the secondary node. If that is done, the global IP > (carp?) switch to the new node. > > It works with HA-Storage from Solaris 10, but the license are too > expensive on none sun-hardware :-/ for our university. > > Any solutions? FreeBSD + ZFS + HAST + CARP + devd will do what you want. You create a separate hast device for each physical harddrive in the system. That "mirrors" the drives between the two servers. Then you create the ZFS pool on top of the hast devices (use /dev/hast/* instead of /dev/da*). Then you configure CARP to provide the shared virtual IP between the two systems. Configure your iSCSI setup to use this IP. Then you write some scripts to handle the orderly tear down of the ZFS pool on one system, and to handle the orderly importing of the pool on the other system. And you hook those scripts into devd, so that when CARP advertises that it is switching which system is master, then ZFS and iSCSI switches with it. Michael Lucas took some scripts I wrote to do the above and made them a better. You can find a lot of information on doing the above here: http://blather.michaelwlucas.com/?p=241 I've used the above in a VM test setup using a ZFS pool with one raidz1 vdev and iSCSI. Works nicely. You have to make sure your iSCSI clients can handle a small window of inaccessibility while the ZFS pool imports on the slave system. We're planning on moving this to real hardware (24-hot swap drive bays in each server) as soon as it arrives (hopefully next week). -- Freddie Cash fjwcash@gmail.com