Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 19 Jul 1996 16:33:12 -0700 (PDT)
From:      asami@cs.berkeley.edu (Satoshi Asami)
To:        njensen@salsa.habaneros.com
Cc:        questions@freebsd.org
Subject:   Re: 2.1.5R and ccd questions
Message-ID:  <199607192333.QAA13479@baloon.mimi.com>
In-Reply-To: <01BB74CA.1F813D80@jalapeno.habaneros.com> (njensen@salsa.habaneros.com)

next in thread | previous in thread | raw e-mail | index | archive | help
 * Just finished installing 2.1.5R (went very smoothly, thanks
 * Team FreeBSD!) and thought I'd try disk mirroring using
 * ccd. I want to mirror the contents of my entire file system
 * so that I can reboot from one of the disk if the other fails.
 * 
 * I have read the documentation; ccd home page and man
 * pages and am still unclear on exactly what I need to do.
 * 
 * I have two idential SCSI drives, sd0 & sd1. Right now the OS
 * is installed on sd0 and I have done nothing to sd1.

I'm afraid you are somewhat misunderstanding the way ccd mirroring
works.  Despite the name, ccd actually concatenates disk partitions,
not disks themselves.  The result of concatenation is a meta-device
that looks like a disk, which you can further partition up by editing
the disklabel.

As far as I can tell, there are two ways to accomplish what you want
to do.  One is to mirror every partition you want as a separate ccd,
and the other is to first create a mega-partition that is the size of
everything except root & swap (more on this later) and divide this
thing up into pieces.

I'll give you an example on how to do the latter.  The strategy here
is to first create a 1-disk ccd on sd1, partition it up, copy
everything you need onto it from sd0, then set up a 2-disk ccd with
mirroring using both disks.

Beware, I'm not sure at all if this will work, so make a backup of
your system before you try this!

 * 1. recompile a new kernel with 'pseudo-device ccd 4' ?

Yes.  The ccd lkm unfortunately doesn't work right now.

 * 2. use dd to copy the contents of sd0 to sd1? (do I need
 * to use disklabel,partition first?)

If we were just interested in copying things from one disk to another
identical one, I'd say "dd if=/dev/rsd0 of=/dev/rsd1" should probably
work, although I have never copied anything larger than a single
partition myself.

However, there is a little problem here: a filesystem on a regular
partition is not the same as that on a ccd on the same partition.  The 
ccd partition needs its own disklabel and such, so the start of the
filesystem is shifted back somewhat.

What I recommend would be like:

(1) dd if=/dev/rsd0 of=/dev/rsd1 bs=512 count=100

This should copy the fdisk table and disklabel to sd1.  The number
"100" is arbirtary, I'm not sure how many is needed here.

(2) disklabel -e sd1

Edit the disklabel, subtract the size of "c" from the offset of "e"
(or "d" or whatever the first partition after "c" is), and use that as
the size of the new "e" partition.

For instance, if your partition table looked like this:

7 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:    65536        0    4.2BSD     1024  8192     0   # (Cyl.    0 - 4*)
  b:   262144    65536      swap                        # (Cyl.    4*- 20*)
  c:  4199697        0    unused        0     0         # (Cyl.    0 - 261*)
  e:    32768   327680    4.2BSD     1024  8192     0   # (Cyl.   20*- 22*)
  f:   196608   360448    4.2BSD     1024  8192     0   # (Cyl.   22*- 34*)
  g:  3642641   557056    4.2BSD     1024  8192     0   # (Cyl.   34*- 261*)

You should change it to:

5 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:    65536        0    4.2BSD     1024  8192     0   # (Cyl.    0 - 4*)
  b:   262144    65536      swap                        # (Cyl.    4*- 20*)
  c:  4199697        0    unused        0     0         # (Cyl.    0 - 261*)
  e:  3872017   327680    4.2BSD     1024  8192     0   # (Cyl.   20*- 22*)
      ^^^^^^^
       =  4199697 - 327680

(The number of partitions is actually the index of the last
partition.)  Also, don't worry about the cylinder numbers, they are
comments and disklabel will fix them up for you.

 * 3. create the /etc/ccd.conf file with CCDF_MIRROR flag?

Yes.  This is what it eventually should look like:

ccd0 128 CCDF_MIRROR /dev/sd0e /dev/sd1e
     ^^^
The interleave size shouldn't really matter, we are mirroring only two 
disks....

 * 4. now run ccdconfig, creating device ccd0?

ccdconfig won't create the devices.  Use /dev/MAKEDEV to create them.
"sh ./MAKEDEV ccd0" will create /dev/ccd0* and /dev/rccd0*, etc.
ccdconfig is automatically run from /etc/rc if you have a
/etc/ccd.conf file on your system.

Now, this is what you need to do now.  Remember we need to copy
everything onto the 1-disk ccd on sd1 first.

(1) create /etc/ccd.conf with the entry

  ccd0 128 0 /dev/sd1e

(2) create a 1-disk ccd

  ccdconfig -Cv

(3) edit the disklabel

  disklabel -wr ccd0 auto
  disklabel -e ccd0

Now, remember how the disklabel looked like before?

#        size   offset    fstype   [fsize bsize bps/cpg]
...
  e:    32768   327680    4.2BSD     1024  8192     0   # (Cyl.   20*- 22*)
  f:   196608   360448    4.2BSD     1024  8192     0   # (Cyl.   22*- 34*)
  g:  3642641   557056    4.2BSD     1024  8192     0   # (Cyl.   34*- 261*)

Take these numbers, subtract "327680" from all the offsets, and write
that as the new disklabel:

#        size   offset    fstype   [fsize bsize bps/cpg]
...
  e:    32768        0    4.2BSD     1024  8192     0   # (Cyl.   20*- 22*)
  f:   196608    32768    4.2BSD     1024  8192     0   # (Cyl.   22*- 34*)
  g:  3642641   229376    4.2BSD     1024  8192     0   # (Cyl.   34*- 261*)

You will also need to decrement the size of the last partition to make
the whole thing fit into the size of "c".  I'm not exactly sure what
it's going to be, it will be a few dozen sectors less than the 3872017
we gave as the size of the whole thing.  So, make sure that size(g) +
offset(g) = size(c).

(4) Create filesystems

  newfs /dev/ccd0e
  newfs /dev/ccd0f
  newfs /dev/ccd0g

(5) Mount them

Assuming these three partitions are mounted as /var, /usr and /a on sd0:

  mkdir /mnt/var /mnt/usr /mnt/a
  mount -o async /dev/ccd0e /mnt/var
  mount -o async /dev/ccd0f /mnt/usr
  mount -o async /dev/ccd0g /mnt/a

The "-o async" mounts the drives asynchronously, which is dangerous
for normal use but should be ok for this situation.  It's much faster.

(6) Copy the stuff over

  cd /var; find -x . | cpio -pmV /mnt/var
  cd /usr; find -x . | cpio -pmV /mnt/usr
  cd /a; find -x . | cpio -pmV /mnt/a

 * 5. is ccd0 now my device with the original filesystem installed,
 * except now I am mirrored? 

Not yet. :)  Now we need to boot with root/swap from sd0 and the rest from
ccd0 (sd1), and then copy the stuff back to sd0.  It will be like
this:

(1) vi /etc/fstab

  Change "/dev/sd0e" to "/dev/ccd0e", etc.

Don't touch the references to sd0a and sd0b!  Actually, you can add a
new line so that it says

  /dev/sd0b               none                    swap    sw 0 0
  /dev/sd1b               none                    swap    sw 0 0

if you want to use the "b" partition of both disks as swap.

(2) reboot

The system should now come up with something like this:

% mount
/dev/sd0a on / (local)
/dev/ccd0e on /var (local)
/dev/ccd0f on /usr (local)
/dev/ccd0g on /a (local)
procfs on /proc (local)

In addition, "pstat -s" will show both sd0b and sd1b if you added sd1b 
as swap.

(3) edit sd0's disklabel

Do a "disklabel -e sd0" and make it look identical to sd1.  Now it
should have sd0a, sd0b, sd0c (the whole disk) and one big partition
(sd0e in our example).

(4) copy stuff over to sd0

  dd if=/dev/rsd1e of=/dev/rsd0e bs=65536

(5) vi /etc/ccd.conf

Change it to

  ccd0 128 CCDF_MIRROR /dev/sd0e /dev/sd1e

(6) reboot

Pray it works. ;)

 * 6. by default, I think the system will boot from sd0. If sd0 fails, 
 * how do I get the system to boot from sd1, will it automatically 
 * check for a valid system there?

Unfortunately, I don't think you can boot from a concatenated disk.
The ccdconfig utility is run from /etc/rc, after the root partition is
mounted (read-only), so any filesystem other than root should work
fine.  (You can even swap to a ccd if you want, the "swapon -a" is
after ccdconfig in /etc/rc. ;)

Of course you probably want to keep a copy of the root partition as
well, so maybe you can run a find -x | cpio from cron or something.

I think the easiest way to be able to boot from sd1 when sd0 goes bad
would be to install the boot manager.  That way you can select "disk
2" to boot from the second disk easily.  Assuming you have maintained
a copy of sd0a in sd1a, you should be able to boot from there.  Of
course, if sd0 goes totally bad and you have to take it out, sd1 will
now become sd0 so it will boot from there. ;)

Beware, ccd doesn't automatically check if a disk is valid, so you
will have to boot the system with -s and edit the ccd.conf file to use
only one disk if one of the two goes bad.  Also, if you did the find 
-x | cpio from cron, fstab will say root is sd0a so if the (bad) sd0
is still in the system and you are booting from sd1, you have to
change it to sd1a (and take out the sd0b reference as well).

 * 7. how do I test if ccd is working?

Write something to /usr/tmp, and see lights on both disks flash. :)

 * Thanks in advance for your help. I will compile any responses I
 * get for possible inclusion in the handbook.

Well, I'm not sure if this was the kind of answer you wanted, but this
is what I *think* should work.  Again, no guarantees, and if you don't
understand the phrase "boot the system with -s", you may not want to
go this far. ;)

Satoshi



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199607192333.QAA13479>