Date: Mon, 29 Oct 2007 16:05:37 -0200 From: Felipe Neuwald <felipe@neuwald.biz> To: Kevin Thompson <antiduh@csh.rit.edu> Cc: freebsd-geom@freebsd.org Subject: Re: Raid 0 + 1 Message-ID: <472620F1.90807@neuwald.biz> In-Reply-To: <4414.147.177.192.113.1193678595.squirrel@angst.csh.rit.edu> References: <4414.147.177.192.113.1193678595.squirrel@angst.csh.rit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Kevin, A bootable 0+1 RAID isn't my idea, cause I have one small disk to system files, but I'll try use your documentation to make my 0+1 RAID. I have one blog to put these documentation, if it's ok to you, please take a look at ontheroadbrother.blogspot.com . Cheers, Felipe Neuwald Kevin Thompson escreveu: >> Hi Folks, >> >> I talked with my customer, and we decided to implement a 0 + 1 RAID, >> with 4 disks of 250Gb each. >> >> Here is how my RAID is working now: >> > > I have some rough instructions I wrote up about doing a bootable 0+1 Geom > RAID, in wiki format. I don't have a public place to post them, so I've > just copied them below. Feedback is welcome. > > Once you get it setup, I would highly recommend that you experiment with > the setup before relying on it - make sure you know how to handle failed > disk replacements, etc. VMWare/camcontrol/atacontrol is very handy in this > regard. > > --Kevin Thompson > > > > > ==Introduction== > > This article describes one method to build a RAID 0+1 array with 4 logical > disks using the FreeBSD GEOM framework, and being able to boot off of such > an array. > > Building a GEOM mirror and being able to directly boot off of it is a > relatively simple task - this is due to the fact that when mirroring is > done, all FreeBSD slices maintain their disk-level structure - GEOM stores > its metadata at the end of the disk/partition/etc. Most importantly, in > such a configuration the MBR and boot file system is untouched. > > The latter is not the case in RAID 0, RAID 0+1, or RAID 1+0 setups. > > In striping configurations, the boot filesystem is interleaved between the > various disks (usually two), and as such, the filesystem appears to be > corrupt if read from only one disk. > > Since the MBR program on the boot disk is incapable of understanding this > physical block layout, it is unable to find the kernel in order to start > FreeBSD. > > However, should the MBR program be able to read the boot filesystem, it > can then load the kernel and related kernel modules, which would then > allow the system to 'boot' from the raidset. > > Technically speaking, the only file system that has to remain untouched is > the /boot file system. root (/), /usr, /var, /tmp, et cetera may all then > be mounted from a raidset. > > In this example, for simplicity's sake, I protect all of the typical root > file system, not just /boot. A more enterprising user may want to modify > their approach. > > ==Instructions== > Boot with a FreeBSD install disk, then start the fixit console. Next > you'll need to clean up the environment (the fixit console needs a little > updating) > ln -s /dist/boot/kernel /boot/kernel > ln -s /dist/lib /lib > EDITOR=/mnt2/usr/bin/vi; export EDITOR > PATH=$PATH':'/mnt2/sbin':'/mnt2/usr/bin':'/mnt2/usr/sbin > export PATH > > > Now load the kernel modules for the geom modules we're going to be using. > glabel load > gstripe load > gmirror load > > Next, we're going to label each of the drives. The 'da0' drive name is > typically assigned by the bios on boot, by way of some sort of metric such > as chain location. The geom label module writes a fixed label to each > drive, so that no matter where each drive goes, or if new devices are > inserted and the numbering is reordered, the raid set will always work the > same. > glabel label geom0 da0 > glabel label geom1 da1 > glabel label geom2 da2 > glabel label geom3 da3 > > Now install a mbr and basic partioning on each drive. This will create a > single partition taking the entire drive for each drive. > fdisk -vBI /dev/label/geom0 > fdisk -vBI /dev/label/geom1 > fdisk -vBI /dev/label/geom2 > fdisk -vBI /dev/label/geom3 > > On the first disk slice of the first drive, install a simple disk label > and bootstrap code. > bsdlabel -wB /dev/label/geom0s1 > > Now edit the generic label on that disk, setting it as you please. 'a' is > commonly root, 'b' the swap partition and 'd' the rest. Don't create any > more partitions other than a, b and d (d will be used as the provider for > a future geom consumer). > > slave(/u9/antiduh) # bsdlabel -e /dev/label/geom0s1 > # size offset fstype [fsize bsize bps/cpg] > a: 500M 16 4.2BSD > b: 500M * swap > c: # leave as is > d: * * 4.2BSD > > The '*' for size means use whatever is left, and the '*' for offset means > use the next logical offset. > > Once you're finished labeling the first drive, write the label for the > drive out to a file, then use it to initialize the other three disks: > bsdlabel /dev/label/geom0s1 > /file > bsdlabel -R /dev/label/geom1s1 /file > bsdlabel -R /dev/label/geom2s1 /file > bsdlabel -R /dev/label/geom3s1 /file > > > Now, create a GEOM mirror out of the 'a' partition of each drive. This > will eventually be the root partition. The new device will be called > '''boot''', and will enumerate in FreeBSD as '''/dev/mirror/boot''' > gmirror label -vh boot /dev/label/geom0s1a /dev/label/geom1s1a > /dev/label/geom2s1a /dev/label/geom3s1a > > Now we're going to pull together the d partion on each drive, create > pairwise stripes, then mirror the new stripes together. > gstripe label -vh -s 131072 st0 /dev/label/geom0s1d /dev/geom1s1d > gstripe label -vh -s 131072 st1 /dev/label/geom2s1d /dev/geom3s1d > > We should now have two new devices '''/dev/stripe/st0''' and > '''/dev/stripe/st1'''. Now mirror those two devices to create our final > device that will next be used for the rest of our filesystems: > gmirror label -vh gm0 /dev/stripe/st0 /dev/stripe/st1 > > We now have our final device '''/dev/mirror/gm0''' that is going to serve > as the base for our regular filesystems. > > Create a basic slicing of the new disk, then edit to taste. Note that we > don't fdisk the raw gm0 device - we create slices directly on the raw > device. > slave(/u9/antiduh) # bsdlabel -wB /dev/mirror/gm0 > slave(/u9/antiduh) # bsdlabel -e /dev/mirror/gm0 > # size offset fstype [fsize bsize bps/cpg] > a: 1 16 unused # that is just a bare '1', not '1M'. This is to get > around a bug in bsdlabel > c: #-- leave as is -- > e: 1000M * 4.2BSD # /var > f: 500M * 4.2BSD # /tmp > d: * * 4.2BSD # /usr, which just gets the rest > > Now create our filesystems on associated slices (the -U option enables > soft updates) > newfs /dev/mirror/boot > newfs -U /dev/mirror/gm0d > newfs -U /dev/mirror/gm0e > newfs -U /dev/mirror/gm0f > > Now we finally get down to mounting the disks, installing the OS on it, > setting up the OS to boot, and finally booting. > > Mount the root disk as /mnt > mount /dev/mirror/boot /mnt > > Create mount points for our other file systems, then mount them: > mkdir /mnt/usr > mkdir /mnt/var > mkdir /mnt/tmp > > mount /dev/mirror/gm0d /mnt/usr > mount /dev/mirror/gm0e /mnt/var > mount /dev/mirror/gm0f /mnt/tmp > > Now we're going to do a install the fun way: > DESTDIR=/mnt > export DESTDIR > cd /dist/6.2-RELEASE/base; ./install.sh > cd ../ports; ./install.sh > cd ../manpages; ./install.sh > cd ../kernels; ./install.sh GENERIC > mv /mnt/boot/GENERIC/* /mnt/boot/kernel > > > You now have a basic, blank, sorta bootable unconfigured FreeBSD install > on the machine. > > Setup the boot loader to load the needed kernel modules, so that we can > mount root from the mirror. > echo 'geom_label_load="YES"' >> /mnt/boot/loader.conf > echo 'geom_stripe_load="YES"' >> /mnt/boot/loader.conf > echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf > > Set a kernel option to use more memory but make the stripe layer faster, > otherwise stripes are unbearably slow. > echo 'kern.geom.stripe.fast=1"' >> /mnt/boot/loader.conf > > Now setup our initial fstab file: > echo '/dev/label/geom0s1b none swap sw 0 0' >> /mnt/etc/fstab > echo '/dev/label/geom1s1b none swap sw 0 0' >> /mnt/etc/fstab > echo '/dev/label/geom2s1b none swap sw 0 0' >> /mnt/etc/fstab > echo '/dev/label/geom3s1b none swap sw 0 0' >> /mnt/etc/fstab > echo '/dev/mirror/boot / ufs rw 1 1' >> /mnt/etc/fstab > echo '/dev/mirror/gm0d /usr ufs rw 2 2' >> /mnt/etc/fstab > echo '/dev/mirror/gm0e /var ufs rw 2 2' >> /mnt/etc/fstab > echo '/dev/mirror/gm0f /tmp ufs rw 2 2' >> /mnt/etc/fstab > > Reboot and the machine should start the raid sets automatically and mount > off of the raid array automatically. Keep that install CD handy in case > you messed up. > > Login, change the root password, setup rc.conf (hostname, interfaces, ssh, > linux binary compat... ), start installing stuff, etc. > > Enjoy. > > ==Block Diagram== > <code> > da0 --label-->geom0 > da1 --label-->geom1 > da2 --label-->geom2 > da3 --label-->geom3 > > label/geom0s1a --|--mirror-->/dev/mirror/boot > label/geom1s1a --| > label/geom2s1a --| > label/geom3s1a --| > > label/geom0s1d --|--stripe-->/dev/stripe/st0 --|--mirror-->/dev/mirror/gm0 > label/geom1s1d --| | > | > label/geom2s1d --|--stripe-->/dev/stripe/st1 --| > label/geom3s1d --| > > mirror/boot --> / > > mirror/gm0s1d --> /usr > mirror/gm0s1e --> /var > mirror/gm0s1f --> /tmp > > label/geom0s1b --> swap > label/geom1s1b --> swap > label/geom2s1b --> swap > label/geom3s1b --> swap > </code> > > > > > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?472620F1.90807>