Date: Thu, 19 May 2011 16:22:57 -0700 From: Freddie Cash <fjwcash@gmail.com> To: Pawel Jakub Dawidek <pjd@freebsd.org> Cc: freebsd-fs@freebsd.org Subject: Re: HAST + ZFS self healing? Hot spares? Message-ID: <BANLkTi=1psNnEOFxD1YEmuNAHRDyXBdBfw@mail.gmail.com> In-Reply-To: <20110519230921.GF2100@garage.freebsd.pl> References: <85EC77D3-116E-43B0-BFF1-AE1BD71B5CE9@itassistans.se> <20110519181436.GB2100@garage.freebsd.pl> <4DD5A1CF.70807@itassistans.se> <20110519230921.GF2100@garage.freebsd.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, May 19, 2011 at 4:09 PM, Pawel Jakub Dawidek <pjd@freebsd.org>
wrote:
> On Fri, May 20, 2011 at 01:03:43AM +0200, Per von Zweigbergk wrote:
>> Very well, that is how failures are handled. But how do we *recover*
>> from a disk failure? Without taking the entire server down that is.
>
> HAST opens local disk only when changing role to primary or changing
> role to secondary and accepting connection from primary.
> If your disk fails, switch to init for that HAST device, replace you
> disk, call 'hastctl create <resource>' and switch back to primary or
> secondary.
>
>> I already know how to deal with my HBA to hot-add and hot-remove
>> devices. And how to deal with hardware failures on the *secondary*
>> node seems fairly straightforward, after all, it doesn't really
>> matter if the mirroring becomes degraded for a few seconds while I
>> futz around with restarting hastd and such. The primary sees the
>> secondary disappear a few seconds, when it comes back, it will just
>> truck all of the dirty data over. Big deal.
>
> You don't need to restart hastd or stop secondary. Just use hastctl to
> change role to init for the failing resource.
This process works exceedingly well. Just went through it a week or so
ago. You just need to think in layers, the way GEOM works:
Non-HAST setup HAST setup
------------------ ------------------
<disk> <disk>
<controller> <controller>
<glabel, gpt, etc> <glabel, gpt, etc>
<zfs> <hast>
<zfs>
The non-HAST process for replacing a disk in a ZFS pool is:
- zpool offline poolname diskname
- remove dead disk
- insert new disk
- partition, label, etc as needed
- zpool replace poolname olddisk newdisk
- wait for resilver to complete
With HAST, there's only a couple of small changes needed:
- zpool offline poolname diskname <-- removes the /dev/hast node
from the pool
- hastctl role init diskname <-- removes the /dev/hast node
- remove dead disk
- insert new disk
- partition, label, etc as needed
- hastctl role create diskname <-- creates the hast resource
- hastctl role primary diskname <-- creates the new /dev/hast node
- zpool replace poolname olddisk newdisk <-- adds the /dev/hast node to
pool
- wait for resilver to complete
The downside to this setup is that the data on the disk in the secondary
node is lost, as the resilver of the disk on the primary node recreates all
the data on the secondary node. But, at least then you know the data is
good on both disks in the HAST resource.
--
Freddie Cash
fjwcash@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BANLkTi=1psNnEOFxD1YEmuNAHRDyXBdBfw>