From owner-freebsd-fs@FreeBSD.ORG Thu Jun 6 20:06:52 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7FF1E9F5 for ; Thu, 6 Jun 2013 20:06:52 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-pd0-f178.google.com (mail-pd0-f178.google.com [209.85.192.178]) by mx1.freebsd.org (Postfix) with ESMTP id 5D50E1159 for ; Thu, 6 Jun 2013 20:06:52 +0000 (UTC) Received: by mail-pd0-f178.google.com with SMTP id w16so3774693pde.23 for ; Thu, 06 Jun 2013 13:06:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ipXIycMyF9wva91UXUmtgcU6cP6NNrWpdxJIIRXrC7A=; b=HEEEQiOVz758KZ5/+amCBLETLB8I9HbzDt/6PXGqFB+BhqNToByDYdGzukUcFNTLYp WgVPab5MyRNtqrIacDTzapgjudJ/T9cFgFWwMByT6y9Ek0/0EBtQhMfzGtB6uXRvuVrw J/vBVLbdzLPW9OxQ+OIJdKvhF6+hLJUgx4Eh0hvh8xnJiWp9rXrF0ysi8XBhqY7yXHzJ PBhFgNJGESZjlu6NJCWBNdav/Et+irVS9K3Gva0/ATaZJQ8g5ypVnDCleKmMzNl3rmEd BMiUUl+nI6sy40F/H3dSnjo0SH6eVfy58bHL8sS2A/41eiXpoPpzI5XUQlsuDlMX06ze 2I0w== MIME-Version: 1.0 X-Received: by 10.68.160.132 with SMTP id xk4mr40135226pbb.37.1370549205940; Thu, 06 Jun 2013 13:06:45 -0700 (PDT) Received: by 10.66.160.66 with HTTP; Thu, 6 Jun 2013 13:06:45 -0700 (PDT) In-Reply-To: References: Date: Thu, 6 Jun 2013 16:06:45 -0400 Message-ID: Subject: Re: zpool export/import on failover - The pool metadata is corrupted From: Outback Dingo To: mxb Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 20:06:52 -0000 On Thu, Jun 6, 2013 at 3:24 PM, mxb wrote: > > Hello list, > > I have two-head ZFS setup with external disk enclosure over SAS expander. > This is a failover setup with CARP and devd triggering spool export/import. > One of two nodes is preferred master. > > Then master is rebooted, devd kicks in as of CARP becomes master and the > second node picks up ZFS-disks from external enclosure. > Then master comes back, CARP becomes master, devd kicks in and pool gets > exported from the second node and imported on the first one. > > However, I have experienced metadata corruption several times with this > setup. > Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC is both > local and external - da1,da2, da13s2, da14s2 > > root@nfs2:/root # zpool import > pool: jbod > id: 17635654860276652744 > state: FAULTED > status: The pool metadata is corrupted. > action: The pool cannot be imported due to damaged devices or data. > see: http://illumos.org/msg/ZFS-8000-72 > config: > > jbod FAULTED corrupted data > raidz3-0 ONLINE > da3 ONLINE > da4 ONLINE > da5 ONLINE > da6 ONLINE > da7 ONLINE > da8 ONLINE > da9 ONLINE > da10 ONLINE > da11 ONLINE > da12 ONLINE > cache > da1 > da2 > da13s2 > da14s2 > logs > mirror-1 ONLINE > da13s1 ONLINE > da14s1 ONLINE > > Any ideas what is going on? > Best case scenerio, both nodes tried to import the disks simultaneously, split brain condition, or disks appear out of order and no using labels, we had a similar situation with geom_multipath, theres no quorum disks knowledge yet for FreeBSD using zfs in this configuration, we ran it similarly for a while, until we realized through research it was bad karma to let carp and devd control nodes without a fool proof way to be sure nodes were ready to export/import. though you did state "Then master comes back, CARP becomes master, devd kicks in and pool gets exported from the second node and imported on the first one." be nice to see how you managed that with scripts... even if both nodes booted simultaneously their both going to "fight" for master and try to import that pool. > //mxb > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >