Date: Thu, 07 Mar 2013 14:07:04 -0800 From: Cody Ritts <cr@caltel.com> To: freebsd-doc@freebsd.org Subject: Wiki / ZFS Booting Message-ID: <51390F88.3070101@caltel.com>
next in thread | raw e-mail | index | archive | help
Hello all,
I have some suggestions for the wiki regarding root on zfs.
In short, I think it would be EXTREMELY helpful to consolidate all of
the pages into two.
1. Booting to ZFS
2. Advanced ZFS partitioning
The long story is that I have just spent several days documenting my
efforts to get multiple systems to boot from ZFS. First, thank you to
all who have contributed to the wiki, I would not have been able to get
it done without you.
That being said these are 11 pages I have found on the wiki:
/RootOnZFS, /RootOnZFS/UFSBoot, /RootOnZFS/ZFSBootPartition,
/RootOnZFS/ZFSBootSlice, /RootOnZFS/GPTZFSBoot,
/RootOnZFS/GPTZFSBoot/Mirror, /RootOnZFS/GPTZFSBoot/RAIDZ1,
/RootOnZFS/GPTZFSBoot/RAIDZ2, /RootOnZFS/GPTZFSBoot/9.0-RELEASE,
/ZFSOnRoot, /ZFSOnRootWithZFSboot
I have read all of those, plus another half dozen on the internet. While
the steps are largely the same, they all seem to have their own little
differences. I have copied and pasted the commands from more than one
of them into virtual machines, and they would not boot.
My overall experience was extremely draining. I expected that when I am
copying and pasting commands from a wiki, and it does not boot, the
problem is not with the commands. When there are different commands
being run on one example from the other, I have to go research which one
is right. I spent a lot of time chasing my tail.
I don't want to just complain that it should be better, I would like to
share the internal documentation that I put together from all of the web
pages, man pages, and server testing.
I am not 100% certain I am doing everything the "right" way, but I am
100% CERTAIN that I have COPY/PASTED each and every one of these
commands into a FreeBSD-9.1-RELEASE LiveCD shell and then successfully
booted.
These commands cover all of the information that I needed to know that
was spread out across the internet. Hopefully someone can find this
information useful who has the time, inclination, and write access to
the wiki.
Cheers,
Cody
###### Boot to LiveCD on 9.1-RELEASE media ######
## -- I like copy/paste accuracy, so I enable sshd on the liveCD
dhclient em0
mkdir /tmp/etc
mount_unionfs /tmp/etc /etc
sed -i '' 's/#PermitRootLogin no/PermitRootLogin yes/g' /etc/ssh/sshd_config
service sshd onestart
passwd
###### GPT INSTALL ######
gpart create -s gpt ada0
gpart add -s 122 -t freebsd-boot -l boot0 ada0
gpart add -s 2048M -t freebsd-swap -l swap0 ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot
/dev/gpt/disk0
###### MBR INSTALL ######
gpart create -s mbr ada0
gpart add -t freebsd ada0
gpart create -s bsd ada0s1
gpart add -s 51G -t freebsd-zfs ada0s1
gpart add -s 8G -t freebsd-swap ada0s1
glabel label disk0 /dev/ada0s1a
glabel label swap /dev/ada0s1b
gpart set -a active -i 1 ada0
gpart bootcode -b /boot/mbr ada0
dd if=/boot/zfsboot of=/dev/ada0s1 count=1
dd if=/boot/zfsboot of=/dev/ada0s1a skip=1 seek=1024
zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot
/dev/label/disk0
###### ZRAID/MIRROR GPT INSTALL ######
gpart create -s gpt ada0
gpart add -s 122 -t freebsd-boot -l boot0 ada0
gpart add -s 2048M -t freebsd-swap -l swap0 ada0
gpart add -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
gpart create -s gpt ada1
gpart add -s 122 -t freebsd-boot -l boot1 ada1
gpart add -s 2048M -t freebsd-swap -l swap1 ada1
gpart add -t freebsd-zfs -l disk1 ada1
## -- repeat for adaN
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
gmirror label -b load -F swap /dev/gpt/swap0 /dev/gpt/swap1
zpool create -R /mnt -O canmount=off -o cachefile=/tmp/zpool.cache zroot
mirror /dev/gpt/disk0 /dev/gpt/disk1
## -- zpool <options> create zroot [mirror|raidz1|raidz2|raidz3]
/dev/gpt/disk0 ... /dev/gpt/diskN
###### COMMON INSTALL COMMANDS #########
zfs set mountpoint=/zroot zroot
zfs set checksum=fletcher4 zroot
zfs create -o mountpoint=legacy zroot/ROOT
mount -t zfs zroot/ROOT /mnt
tar -C /mnt -xppf /usr/freebsd-dist/base.txz
tar -C /mnt -xppf /usr/freebsd-dist/kernel.txz
zpool set bootfs=zroot/ROOT zroot
zfs set freebsd:boot-environment=1 zroot/ROOT
cp /tmp/zpool.cache /mnt/boot/zfs/
echo 'hostname="host"' >> /mnt/etc/rc.conf
echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot/ROOT"' >> /mnt/boot/loader.conf
###### GPT INSTALL ######
cat << EOF > /mnt/etc/fstab
# Device Mountpoint FStype Options
Dump Pass#
/dev/gpt/swap0 none swap sw 0 0
EOF
###### MBR INSTALL ######
cat << EOF > /mnt/etc/fstab
# Device Mountpoint FStype Options
Dump Pass#
/dev/label/swap none swap sw 0 0
EOF
###### ZRAID/MIRROR GPT INSTALL ######
echo 'geom_mirror_load="YES"' >> /mnt/boot/loader.conf
cat << EOF > /mnt/etc/fstab
# Device Mountpoint FStype Options
Dump Pass#
/dev/mirror/swap none swap sw
0 0
EOF
###### CHROOT INSTALL COMMANDS ######
chroot /mnt
passwd root
tzsetup
cd /etc/mail
make aliases
exit
reboot
###### DISCUSSION ######
## -- notes on swap --
swap can go on ZFS filesystem but can no longer be used as a dump device.
## -- mountpoint=legacy for zroot/ROOT
release/9.1.0/tools/tools/zfsboottest/zfsboottest.sh
"Dataset's mountpoint property should be set to 'legacy'"
I would imagine this protects you against importing the
pool on another system, and forgetting the altroot command.
When you are set legacy, you MUST either have:
echo vfs.root.mountfrom="zfs:zroot/ROOT"' >> /mnt/boot/loader.conf
OR
an entry in fstab (I have not tested this)
## -- File system layout --
There are a lot of really cool things you can do with zfs layout
Please see [link to zfs Layout Wiki page]
The only advanced thing we did here is moved the root file system
so that /tmp /usr and /var do not inherit zfs commands from /
## -- notes on zpool.cache --
in 9.1-RELEASE -- without zpool.cache in this configuration, the the
system will NOT boot.
## -- Exporting zroot --
in 9.1-RELEASE -- If you export zroot, your system will not boot.
You MUST boot from liveCD, and re-import it to get it to boot.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51390F88.3070101>