Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 29 Oct 2007 13:23:15 -0400 (EDT)
From:      "Kevin Thompson" <antiduh@csh.rit.edu>
To:        "Felipe Neuwald" <felipe@neuwald.biz>
Cc:        freebsd-geom@freebsd.org
Subject:   Re: Raid 0 + 1
Message-ID:  <4414.147.177.192.113.1193678595.squirrel@angst.csh.rit.edu>

next in thread | raw e-mail | index | archive | help
> Hi Folks,
>
> I talked with my customer, and we decided to implement a 0 + 1 RAID,
> with 4 disks of 250Gb each.
>
> Here is how my RAID is working now:

I have some rough instructions I wrote up about doing a bootable 0+1 Geom
RAID, in wiki format. I don't have a public place to post them, so I've
just copied them below. Feedback is welcome.

Once you get it setup, I would highly recommend that you experiment with
the setup before relying on it - make sure you know how to handle failed
disk replacements, etc. VMWare/camcontrol/atacontrol is very handy in this
regard.

--Kevin Thompson




==Introduction==

This article describes one method to build a RAID 0+1 array with 4 logical
disks using the FreeBSD GEOM framework, and being able to boot off of such
an array.

Building a GEOM mirror and being able to directly boot off of it is a
relatively simple task - this is due to the fact that when mirroring is
done, all FreeBSD slices maintain their disk-level structure - GEOM stores
its metadata at the end of the disk/partition/etc. Most importantly, in
such a configuration the MBR and boot file system is untouched.

The latter is not the case in RAID 0, RAID 0+1, or RAID 1+0 setups.

In striping configurations, the boot filesystem is interleaved between the
various disks (usually two), and as such, the filesystem appears to be
corrupt if read from only one disk.

Since the MBR program on the boot disk is incapable of understanding this
physical block layout, it is unable to find the kernel in order to start
FreeBSD.

However, should the MBR program be able to read the boot filesystem, it
can then load the kernel and related kernel modules, which would then
allow the system to 'boot' from the raidset.

Technically speaking, the only file system that has to remain untouched is
the /boot file system. root (/), /usr, /var, /tmp, et cetera may all then
be mounted from a raidset.

In this example, for simplicity's sake, I protect all of the typical root
file system,  not just /boot. A more enterprising user may want to modify
their approach.

==Instructions==
Boot with a FreeBSD install disk, then start the fixit console. Next
you'll need to clean up the environment (the fixit console needs a little
updating)
 ln -s /dist/boot/kernel /boot/kernel
 ln -s /dist/lib /lib
 EDITOR=/mnt2/usr/bin/vi; export EDITOR
 PATH=$PATH':'/mnt2/sbin':'/mnt2/usr/bin':'/mnt2/usr/sbin
 export PATH


Now load the kernel modules for the geom modules we're going to be using.
 glabel load
 gstripe load
 gmirror load

Next, we're going to label each of the drives. The 'da0' drive name is
typically assigned by the bios on boot, by way of some sort of metric such
as chain location. The geom label module writes a fixed label to each
drive, so that no matter where each drive goes, or if new devices are
inserted and the numbering is reordered, the raid set will always work the
same.
 glabel label geom0 da0
 glabel label geom1 da1
 glabel label geom2 da2
 glabel label geom3 da3

Now install a mbr and basic partioning on each drive. This will create a
single partition taking the entire drive for each drive.
 fdisk -vBI /dev/label/geom0
 fdisk -vBI /dev/label/geom1
 fdisk -vBI /dev/label/geom2
 fdisk -vBI /dev/label/geom3

On the first disk slice of the first drive, install a simple disk label
and bootstrap code.
 bsdlabel -wB /dev/label/geom0s1

Now edit the generic label on that disk, setting it as you please. 'a' is
commonly root, 'b' the swap partition and 'd' the rest. Don't create any
more partitions other than a, b and d (d will be used as the provider for
a future geom consumer).

 slave(/u9/antiduh) # bsdlabel -e /dev/label/geom0s1
 #  size   offset    fstype   [fsize bsize bps/cpg]
 a: 500M   16 	     4.2BSD
 b: 500M   *         swap
 c:                         # leave as is
 d: *      *         4.2BSD

The '*' for size means use whatever is left, and the '*' for offset means
use the next logical offset.

Once you're finished labeling the first drive, write the label for the
drive out to a file, then use it to initialize the other three disks:
 bsdlabel /dev/label/geom0s1 > /file
 bsdlabel -R /dev/label/geom1s1 /file
 bsdlabel -R /dev/label/geom2s1 /file
 bsdlabel -R /dev/label/geom3s1 /file


Now, create a GEOM mirror out of the 'a' partition of each drive. This
will eventually be the root partition. The new device will be called
'''boot''', and will enumerate in FreeBSD as '''/dev/mirror/boot'''
 gmirror label -vh boot /dev/label/geom0s1a /dev/label/geom1s1a
/dev/label/geom2s1a /dev/label/geom3s1a

Now we're going to pull together the d partion on each drive, create
pairwise stripes, then mirror the new stripes together.
 gstripe label -vh -s 131072 st0 /dev/label/geom0s1d /dev/geom1s1d
 gstripe label -vh -s 131072 st1 /dev/label/geom2s1d /dev/geom3s1d

We should now have two new devices '''/dev/stripe/st0''' and
'''/dev/stripe/st1'''. Now mirror those two devices to create our final
device that will next be used for the rest of our filesystems:
 gmirror label -vh gm0 /dev/stripe/st0 /dev/stripe/st1

We now have our final device '''/dev/mirror/gm0''' that is going to serve
as the base for our regular filesystems.

Create a basic slicing of the new disk, then edit to taste. Note that we
don't fdisk the raw gm0 device - we create slices directly on the raw
device.
 slave(/u9/antiduh) # bsdlabel -wB /dev/mirror/gm0
 slave(/u9/antiduh) # bsdlabel -e /dev/mirror/gm0
 #        size   offset    fstype   [fsize bsize bps/cpg]
 a: 1		16	unused	# that is just a bare '1', not '1M'. This is to get
around a bug in bsdlabel
 c: #-- leave as is --
 e: 1000M 	*	4.2BSD	# /var
 f: 500M	*	4.2BSD  # /tmp
 d: *		*	4.2BSD  # /usr, which just gets the rest

Now create our filesystems on associated slices (the -U option enables
soft updates)
 newfs /dev/mirror/boot
 newfs -U /dev/mirror/gm0d
 newfs -U /dev/mirror/gm0e
 newfs -U /dev/mirror/gm0f

Now we finally get down to mounting the disks, installing the OS on it,
setting up the OS to boot, and finally booting.

Mount the root disk as /mnt
 mount /dev/mirror/boot /mnt

Create mount points for our other file systems, then mount them:
 mkdir /mnt/usr
 mkdir /mnt/var
 mkdir /mnt/tmp

 mount /dev/mirror/gm0d /mnt/usr
 mount /dev/mirror/gm0e /mnt/var
 mount /dev/mirror/gm0f /mnt/tmp

Now we're going to do a install the fun way:
 DESTDIR=/mnt
 export DESTDIR
 cd /dist/6.2-RELEASE/base; ./install.sh
 cd ../ports; ./install.sh
 cd ../manpages; ./install.sh
 cd ../kernels; ./install.sh GENERIC
 mv /mnt/boot/GENERIC/* /mnt/boot/kernel


You now have a basic, blank, sorta bootable unconfigured FreeBSD install
on the machine.

Setup the boot loader to load the needed kernel modules, so that we can
mount root from the mirror.
 echo 'geom_label_load="YES"'  >> /mnt/boot/loader.conf
 echo 'geom_stripe_load="YES"' >> /mnt/boot/loader.conf
 echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf

Set a kernel option to use more memory but make the stripe layer faster,
otherwise stripes are unbearably slow.
 echo 'kern.geom.stripe.fast=1"' >> /mnt/boot/loader.conf

Now setup our initial fstab file:
 echo '/dev/label/geom0s1b	none	swap	sw	0	0' >> /mnt/etc/fstab
 echo '/dev/label/geom1s1b	none	swap	sw	0	0' >> /mnt/etc/fstab
 echo '/dev/label/geom2s1b	none	swap	sw	0	0' >> /mnt/etc/fstab
 echo '/dev/label/geom3s1b	none	swap	sw	0	0' >> /mnt/etc/fstab
 echo '/dev/mirror/boot		/	ufs	rw	1 	1' >> /mnt/etc/fstab
 echo '/dev/mirror/gm0d		/usr	ufs	rw	2 	2' >> /mnt/etc/fstab
 echo '/dev/mirror/gm0e		/var	ufs	rw	2 	2' >> /mnt/etc/fstab
 echo '/dev/mirror/gm0f		/tmp	ufs	rw	2 	2' >> /mnt/etc/fstab

Reboot and the machine should start the raid sets automatically and mount
off of the raid array automatically. Keep that install CD handy in case
you messed up.

Login, change the root password, setup rc.conf (hostname, interfaces, ssh,
linux binary compat... ), start installing stuff, etc.

Enjoy.

==Block Diagram==
<code>
 da0 --label-->geom0
 da1 --label-->geom1
 da2 --label-->geom2
 da3 --label-->geom3

 label/geom0s1a --|--mirror-->/dev/mirror/boot
 label/geom1s1a --|
 label/geom2s1a --|
 label/geom3s1a --|

 label/geom0s1d --|--stripe-->/dev/stripe/st0 --|--mirror-->/dev/mirror/gm0
 label/geom1s1d --|                             |
                                                |
 label/geom2s1d --|--stripe-->/dev/stripe/st1 --|
 label/geom3s1d --|

 mirror/boot   --> /

 mirror/gm0s1d --> /usr
 mirror/gm0s1e --> /var
 mirror/gm0s1f --> /tmp

 label/geom0s1b --> swap
 label/geom1s1b --> swap
 label/geom2s1b --> swap
 label/geom3s1b --> swap
</code>







Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4414.147.177.192.113.1193678595.squirrel>