From owner-freebsd-fs@FreeBSD.ORG Thu Jun 6 22:12:43 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 54C98203 for ; Thu, 6 Jun 2013 22:12:43 +0000 (UTC) (envelope-from mxb@alumni.chalmers.se) Received: from mail-la0-x233.google.com (mail-la0-x233.google.com [IPv6:2a00:1450:4010:c03::233]) by mx1.freebsd.org (Postfix) with ESMTP id B6FC71857 for ; Thu, 6 Jun 2013 22:12:42 +0000 (UTC) Received: by mail-la0-f51.google.com with SMTP id fq12so2904883lab.38 for ; Thu, 06 Jun 2013 15:12:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to:x-mailer:x-gm-message-state; bh=Q2zyLrlpSLRt0W+gZ/WvESlZMEpNhyCXzoSM4wpqfhc=; b=PK0X/KW8uQXzYcEEj0fFxW0gQ11w6JMWO+m3gUYNoxJLQ7HxTIu5xULtYUWBTvbAwm 3CDgUGRs3VhFol7zJnEDc7AitIMC/TpOfIFuuTqnMWA+ts2++Jm2E1FCdDcQhLIzxZJV wUyJQ9ziQBQmH9RYueS2iDstxli4o0gjjT4aq38URgxa/HCxn0h8ZAeMyhvvPLpby6zB 8ydK2Viksrb6RFaGPQ9tDyc49JNiZoH0pSaX6uI7hdEoPmimewGWQioMgygUhQhz9jtY Ac93A2xf1Nidr1qE61swhs5vQhbkv2qIeIR55Fb/RMThL1cnqqS012B7iNg9gvAbRs+Q 1LLg== X-Received: by 10.112.144.69 with SMTP id sk5mr821313lbb.64.1370556761613; Thu, 06 Jun 2013 15:12:41 -0700 (PDT) Received: from grey.home.unixconn.com (h-75-17.a183.priv.bahnhof.se. [46.59.75.17]) by mx.google.com with ESMTPSA id f9sm29461076lbf.4.2013.06.06.15.12.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 06 Jun 2013 15:12:40 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: zpool export/import on failover - The pool metadata is corrupted From: mxb In-Reply-To: Date: Fri, 7 Jun 2013 00:12:39 +0200 Message-Id: <016B635E-4EDC-4CDF-AC58-82AC39CBFF56@alumni.chalmers.se> References: To: Outback Dingo X-Mailer: Apple Mail (2.1503) X-Gm-Message-State: ALoCoQkC1MeVAlH42wUcCbWVi2xfAeB3FD1IMBrZsjg8dEShP9Ncfsl7TsYX8GBFFnspoAg9ZAWG Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 22:12:43 -0000 Then MASTER goes down, CARP on the second node goes MASTER (devd.conf, = and script for lifting): root@nfs2:/root # cat /etc/devd.conf notify 30 { match "system" "IFNET"; match "subsystem" "carp0"; match "type" "LINK_UP"; action "/etc/zfs_switch.sh active"; }; notify 30 { match "system" "IFNET"; match "subsystem" "carp0"; match "type" "LINK_DOWN"; action "/etc/zfs_switch.sh backup"; }; root@nfs2:/root # cat /etc/zfs_switch.sh #!/bin/sh DATE=3D`date +%Y%m%d` HOSTNAME=3D`hostname` ZFS_POOL=3D"jbod" case $1 in active) echo "Switching to ACTIVE and importing ZFS" | mail -s = ''$DATE': '$HOSTNAME' switching to ACTIVE' root sleep 10 /sbin/zpool import -f jbod /etc/rc.d/mountd restart /etc/rc.d/nfsd restart ;; backup) echo "Switching to BACKUP and exporting ZFS" | mail -s = ''$DATE': '$HOSTNAME' switching to BACKUP' root /sbin/zpool export jbod /etc/rc.d/mountd restart /etc/rc.d/nfsd restart ;; *) exit 0 ;; esac This works, most of the time, but sometimes I'm forced to re-create = pool. Those machines suppose to go into prod. Loosing pool(and data inside it) stops me from deploy this setup. //mxb On 6 jun 2013, at 22:06, Outback Dingo wrote: >=20 >=20 >=20 > On Thu, Jun 6, 2013 at 3:24 PM, mxb wrote: >=20 > Hello list, >=20 > I have two-head ZFS setup with external disk enclosure over SAS = expander. > This is a failover setup with CARP and devd triggering spool = export/import. > One of two nodes is preferred master. >=20 > Then master is rebooted, devd kicks in as of CARP becomes master and = the second node picks up ZFS-disks from external enclosure. > Then master comes back, CARP becomes master, devd kicks in and pool = gets exported from the second node and imported on the first one. >=20 > However, I have experienced metadata corruption several times with = this setup. > Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC is = both local and external - da1,da2, da13s2, da14s2 >=20 > root@nfs2:/root # zpool import > pool: jbod > id: 17635654860276652744 > state: FAULTED > status: The pool metadata is corrupted. > action: The pool cannot be imported due to damaged devices or data. > see: http://illumos.org/msg/ZFS-8000-72 > config: >=20 > jbod FAULTED corrupted data > raidz3-0 ONLINE > da3 ONLINE > da4 ONLINE > da5 ONLINE > da6 ONLINE > da7 ONLINE > da8 ONLINE > da9 ONLINE > da10 ONLINE > da11 ONLINE > da12 ONLINE > cache > da1 > da2 > da13s2 > da14s2 > logs > mirror-1 ONLINE > da13s1 ONLINE > da14s1 ONLINE >=20 > Any ideas what is going on? >=20 > Best case scenerio, both nodes tried to import the disks = simultaneously, split brain condition, or disks appear out of order and = no using labels, we had a similar situation with geom_multipath, theres = no quorum disks knowledge yet for FreeBSD using zfs in this = configuration, we ran it similarly for a while, until we realized = through research it was bad karma to let carp and devd control nodes = without a fool proof way to be sure nodes were ready to export/import. = though you did state >=20 > "Then master comes back, CARP becomes master, devd kicks in and pool = gets exported from the second node and imported on the first one." be = nice to see how you managed that with scripts... even if both nodes = booted simultaneously their both going to "fight" for master and try to = import that pool. >=20 >=20 > //mxb >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20