From owner-freebsd-stable@FreeBSD.ORG Sun Jun 27 09:16:31 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07783106566C for ; Sun, 27 Jun 2010 09:16:31 +0000 (UTC) (envelope-from matt@bubblegen.co.uk) Received: from relay.pcl-ipout01.plus.net (relay.pcl-ipout01.plus.net [212.159.7.99]) by mx1.freebsd.org (Postfix) with ESMTP id 686148FC1A for ; Sun, 27 Jun 2010 09:16:29 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av0EAHKzJkxUXeb0/2dsb2JhbACDHZwQca1lj3iBKYFfBIEmcgQ Received: from outmx03.plus.net ([84.93.230.244]) by relay.pcl-ipout01.plus.net with ESMTP; 27 Jun 2010 10:16:28 +0100 Received: from bubblegen.plus.com ([80.229.236.194] helo=[192.136.1.18]) by outmx03.plus.net with esmtp (Exim) id 1OSnyW-0007Rn-61; Sun, 27 Jun 2010 10:16:28 +0100 From: Matthew Lear To: Matthew Seaman In-Reply-To: <4C270D9C.3000801@infracaninophile.co.uk> References: <1276889330.2210.44.camel@almscliff.bubblegen.co.uk> <1277155992.1860.3.camel@almscliff.bubblegen.co.uk> <20100622074541.GA71157@icarus.home.lan> <82A96ECD-676C-4A4D-A328-0CFAABD64D50@gid.co.uk> <1277401934.1874.12.camel@almscliff.bubblegen.co.uk> <20100624181535.GA58443@icarus.home.lan> <1277417182.1874.30.camel@almscliff.bubblegen.co.uk> <20100625071644.GA75910@icarus.home.lan> <1277567868.1870.21.camel@almscliff.bubblegen.co.uk> <20100626171251.GA26022@icarus.home.lan> <1277593488.1884.107.camel@almscliff.bubblegen.co.uk> <4C270D9C.3000801@infracaninophile.co.uk> Content-Type: text/plain; charset="UTF-8" Date: Sun, 27 Jun 2010 10:16:27 +0100 Message-ID: <1277630187.1832.2.camel@almscliff.bubblegen.co.uk> Mime-Version: 1.0 X-Mailer: Evolution 2.28.3 Content-Transfer-Encoding: 7bit Cc: Adam Vande More , freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: 7.2-RELEASE-p4, IO errors & RAID1 failure X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jun 2010 09:16:31 -0000 On Sun, 2010-06-27 at 09:36 +0100, Matthew Seaman wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 27/06/2010 24:04:48, Matthew Lear wrote: > > Incidentally, is there a way to easily migrate from a atacontrol created > > array to a gmirror created array? I'm running FreeBSD 8.0 on another > > machine with a gmirror created RAID1 array with no problem whatsoever (I > > chose gmirror as the choice for this machine over atacontrol after > > reading various postings about software RAID under recent releases of > > FreeBSD). I was planning on upgrading the 7.2 machine to 8.0-RC1 anyway > > so if I could easily move to using gmirror then I would. That said, > > atacontrol should (I assume) function correctly with 8.x, shouldn't it, > > or is support of it dwindling somewhat? > > So long as your ataraid setup is a simple mirror, this is actually quite > easy. > > Lets assume you have two identical disks, ad0 and ad2 from which you > create a RAID ar0. To switch over to gmirror based RAID: > > * Edit /etc/fstab changing all references to ar0 to read ad0, so > eg. change: > /dev/ar0s1a / ufs rw 1 1 > to read: > /dev/ad0s1a / ufs rw 1 1 > > and similarly for any other partitions on ar0 (don't forget the > swap area) > > * Edit /boot/loader.conf to load the necessary gmirror kernel modules > add: > geom_mirror_load="YES" > > * Reboot > If necessary, drop into the system BIOS and make sure that which > ever drive corresponding to your ad0 is set as the primary drive > to boot from. > > This should bring your system up on just your ad0 drive -- so no > resilience for a few minutes. > > (Effectively, you're back at the default system on a single > disk layout now) > > * Delete the old ar0 RAID: > # atacontrol delete ar0 > > This should have no effect on the running system as all it does > is remove some meta-data stored on the disk. > > * Configure a new geom mirror device. You need to know the sysctl > magic to let you futz with a mounted drive: > # sysctl kern.geom.debugflags=16 > # gmirror label -v -b load gm0 /dev/ad0 > > (If you're a fan of the classic article on OnLamp > http://onlamp.com/pub/a/bsd/2005/11/10/FreeBSD_Basics.html) which > recommends using round-robin as the balancing algorithm: that's > fine as well, but 'load' as a mechanism wasn't available when > that article was written, and it is generally recognised as > superior nowadays.) > > * Re-edit /etc/fstab to change all references from ad0 to mirror/gm0 > so eg. change: > /dev/ad0s1a / ufs rw 1 1 > to: > /dev/mirror/gm0s1a / ufs rw 1 1 > > and similarly for any other partitions on ad0 > > * Reboot. You should see messages towards the end of the kernel > initialisation sequence saying gmirror raid created successfully. > > * Now add your second drive to the mirror: > # gmirror insert gm0 /dev/ad2 > > * The system will spend some time synchronizing the data from ad0 to > ad0 -- the system can be put back into use immediately if needed, > but it's better to let the resynch finish first, as you won't be > resilient until it has. Also, active disk use will slow down the > synchronisation process. Monitor synchronisation progress by > 'gmirror status gm0' or by running gstat(8). > > * Once everything is synch'd it's all ready for immediate use. > > > How easy is it to upgrade an array to use larger disks - atacontrol or > > gmirror? Feel free to respond with RTFM :-) I suppose one possible > > solution would be to use something like GpartEd (example Linux land > > tool) to grow a certain partition on an array (eg the partition mounted > > on /usr/local). That way both partitions on each of the separate array > > subdisks would be grown transparently since the operation would be > > performed on partition ar0s1 (ie, taken care of by atacontrol / > > gmirror). > > gmirror doesn't really help you with resizing disks or filesystems. > Note the the recipe above assumes you want to mirror the entire contents > of one disk onto another -- which is the usual idea for the system drive > used by a server -- so ideally you want two identical drives. If one of > your drives fails and you need to replace it, but can't get the exact > same model, you will have to use a *bigger* drive -- ie. a larger number > of sectors. Be wary when buying, as disks of different models but with > nominally the same capacity can easily vary by a few thousand sectors. > > Anyhow, it is certainly possible to replace one of the drives in a > gmirror setup with another of much larger capacity, synchronise the > drives, then replace the second drive with a similar larger one and > synchronise back again. If you have hot-swap capability, you can do all > that without downtime. However, you will end up with a system layout > exactly the same size as the original disks. In order to grow the > filesystems to fill the drives, you will have to do all sorts of > high-risk invasive procedures to rewrite partition tables and run > growfs(8) to expand your UFS filesystems to fill the available space. > Note: you can't shrink UFS partitions easily -- generally you have to > backup the data, and then delete and recreate the partition as a smaller > size and restore the data. > > If you want to do this sort of migration from ar0 to a much bigger > gmirror setup, probably the lowest risk, least effort route is to mount > the two new drives on the system, configure a gmirror across the two, > partition and create filesystems on the gmirror, use dump(8) and > restore(8) to copy the filesystems from your live disks, then shutdown, > take out the old ar0 disks and boot from the new gmirror. Remember > you're going to have to fix /etc/fstab on the gmirror before rebooting. > > If you can't fit 4 drives in your system at once, then take out one > drive (which will break the ar0 mirror), replace it with one of your new > disks, build the gmirror on it as above, then shutdown, remove the 2nd > ar0 drive, reboot, add the 2nd gmirror drive and resynch. > > If you really want to be able to resize partitions easily or swap out > the hardware and have the system utilize the increased space without a > great deal of faffing around, then ZFS is your friend. See: > > http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror > > You should be able to adapt those instructions to install a ZFS system > on some fresh drives attached to an existing system which uses > atacontrol or gmirror raid + UFS rather than booting from installation > media, or you can split your original mirror and build half a ZFS mirror > in the space that creates, then resynch disks afterwards. > > It's more work to set up ZFS initially, but once it is done, it just > ticks along happily and it makes any subsequent disk manipulations much > easier and safer. > Excellent info, Matthew. Thank you very much indeed.