Date: Sat, 11 Feb 2012 10:57:11 -0600 From: Trix Farrar <trix@basement.net> To: FreeBSD Xen <freebsd-xen@freebsd.org>, Xen-Users <xen-users@lists.xensource.com> Subject: 9.0-RELEASE PV from scratch on XCP v1.1.0 Message-ID: <20120211165711.GB71096@basement.net>
next in thread | raw e-mail | index | archive | help
--O3RTKUHj+75w1tg5 Content-Type: multipart/mixed; boundary="WYTEVAkct0FjGQmd" Content-Disposition: inline --WYTEVAkct0FjGQmd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Apologies for the re-post. Forgot to sign original. -Trx Title: HOWTO-FreeBSD-on-XCP Author: John D. "Trix" Farrar Date: 2012-02-09 * The Challenge - ParaVirtualized FreeBSD 9.0-RELEASE under XCP 1.1.0 The idea here is to create a PV FreeBSD VM under Xen Cloud Platform (XCP) without starting with an HVM first. The documentation I''ve been able to find on-line is at least a year old and is mostly written for Xen (under Linux or NetBSD) rather than XCP. To make things more interesting, the hardware on which I have XCP installed doesn''t support Hardware Virtualization -- so I can''t just install an HVM and convert it over. A PV under XCP means using pygrub as a bootloader to load the FreeBSD kernel. It seems that pygrub doesn''t know enough about BSD disk slices and partitions to be able to find the kernel on its own, so it will need a patch This solution isn''t perfect, but I think it''s a good start. * The Equipment Yes, my hardware is old/cheap. * Xen Cloud Platform server - XCP v1.1.0 * Build Server - FreeBSD 9.0-RELEASE (32-bit) * NFS Server - anything w/ enough space. Can be the Build Server. (should NOT be the XCP Server) * The Process * Patching pygrub Adrian Chadd (http://wiki.freebsd.org/AdrianChadd/XenHackery) did a great job laying the groundwork for a Xen (not XCP) installation. I also got a lot of help from Adian''s patch to pygrub at http://people.freebsd.org/~adrian/xen/bsd_pygrub/. The problem is that Adrian''s patch "is against the pygrub shipped with the xen-3.0.3-80.el5_3.2 CentOS 5.3 package." Using Adrian''s patch, I wsa able to create a patch against the pygrub shipped with XCP v1.1.0: =3D=3DBEGIN=3D=3D Patch attached. =3D=3D=3DEND=3D=3D=3D =20 * Creating a disk image I mounted an exported filesystem from the NFS server so that I would have a large, sharable space in which to place the disk image. This space is also mounted on the XCP server. In both cases, the NFS filesystem is mounted under /mnt. The following steps are taken on the BUILD host where /usr/src has already been compiled with buiidworld and buildkernel targets. =3D=3DBEGIN=3D=3D export FSIMAGE=3D"FreeBSD-PV.img" truncate -s 10G /mnt/${FSIMAGE} # Create the file where the im= age will reside. mdconfig -a -t vnode -f /mnt/${FSIMAGE} -u 0 # Create the image itself fdisk -BI md0 # Write the DOS partition table # Create an inital BSD Label=20 # For testing, we'll assume one large partition bsdlabel -w -B md0s1 bsdlabel -e md0s1 # Change partition a from 'unused' to '4.2BSD' newfs -O1 /dev/md0s1a # UFS, because pygrub doesn't grok UFS2 # Mount up the image and do the installation mount /dev/md0s1a /media # Perform the installation to the disk image. cd /usr/src make -s DESTDIR=3D/media KERNCONF=3DXEN installworld make -s DESTDIR=3D/media KERNCONF=3DXEN installkernel make -s DESTDIR=3D/media KERNCONF=3DXEN distribution # Set up the ttys(5) file so the VM will use Xen's console. cat >>/media/etc/ttys <<EOF # Xen Console xc0 "/usr/libexec/getty Pc" vt100 on secure EOF # Create fstab(5) so the root (only) filesystem will mount. cat >>/media/etc/fstab <<EOF # Device Mountpoint FStype Options Dump Pass# /dev/xbd0s1a / ufs rw 1 1 EOF # Start and rc.conf(5) file # NOTE: You won't be able to login via SSH until users are added. cat >>/media/etc/rc.conf <<EOF hostname=3D"${VMNAME}" ifconfig_xn0=3D'dhcp' sshd_enable=3D'YES' EOF # BUILD Server - Unmount the filesystem, the image is done. umount /media mdconfig -d -u 0 =3D=3D=3DEND=3D=3D=3D The following section is the script I actually used to create a VM that is more-or-less realistic as a template. =3D=3DBEGIN-Advanced-Option=3D=3D truncate -s 20G /mnt/${FSIMAGE} mdconfig -a -t vnode -f /mnt/${FSIMAGE} -u 0 fdisk -BI md0 bsdlabel -w -B md0s1 cat >>xcp-optionb-bsdlabel.txt <<EOF # /dev/md0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 512MB 63 4.2BSD 0 0 0 b: 512MB * swap =20 c: 41929587 0 unused 0 0 # "raw" part, don't= edit d: 512MB * 4.2BSD 0 0 0 e: 2097152 3145791 4.2BSD 0 0 0 f: 6291456 5242943 4.2BSD 0 0 0 g: 12582912 11534399 4.2BSD 0 0 0 h: 17812276 24117311 4.2BSD 0 0 0 EOF bsdlabel -R md0s1 xcp-optionb-bsdlabel.txt rm xcp-optionb-bsdlabel.txt # root partition has to be UFS1. PyGRUB chokes on UFS2 (fsimage.ufs, actua= lly) newfs -O1 /dev/md0s1a && mount -v /dev/md0s1a /media newfs /dev/md0s1d && mkdir /media/tmp && mount -v /dev/md0s1d /me= dia/tmp newfs /dev/md0s1e && mkdir /media/var && mount -v /dev/md0s1e /me= dia/var newfs /dev/md0s1f && mkdir /media/usr && mount -v /dev/md0s1f /me= dia/usr=20 newfs /dev/md0s1g && mkdir /media/usr/local && mount -v /dev/md0s1g /me= dia/usr/local newfs /dev/md0s1h && mkdir /media/home && mount -v /dev/md0s1h /me= dia/home pushd /usr/src make -s DESTDIR=3D/media KERNCONF=3DXEN installworld installkernel distribu= tion popd cat >>/media/etc/ttys <<EOF # Xen Console xc0 "/usr/libexec/getty Pc" vt100 on secure EOF cat >>/media/etc/fstab <<EOF # Device Mountpoint FStype Options Dump Pass# /dev/xbd0s1a / ufs rw 1 1 /dev/xbd0s1f /usr ufs rw 1 1 /dev/xbd0s1g /usr/local ufs rw 1 1 /dev/xbd0s1d /tmp ufs rw 1 1 /dev/xbd0s1e /var ufs rw 1 1 /dev/xbd0s1h /home ufs rw 1 1 EOF cat >>/media/etc/rc.conf <<EOF hostname=3D"freebsd-9.0" ifconfig_xn0=3D'dhcp' sshd_enable=3D'YES' EOF umount -v /media/home /media/usr/local /media/usr /media/var /media/tmp /me= dia =3D=3D=3DEND-Advanced-Option=3D=3D=3D * Create the Virtual Host I got the bulk of this information from Grant McWilliams'' HOWTO for installing CentOS on an XCP host=20 (http://grantmcwilliams.com/item/563-centos6-on-xcp). =20 All of the next steps are to be carried on at the Local Command Shell prompt on the XCP host. =3D=3DBEGIN=3D=3D VMNAME=3D"FreeBSD-9.0" # Edit this value, each VM must be different # XCP Server - Create the Virtual Disk Image and copy .img file into it. export SR_UUID=3D$(xe sr-list name-label=3D"Local storage" --minimal) export VDIUUID=3D$(xe vdi-create \ sr-uuid=3D${SR_UUID} \ name-label=3D${VMNAME}-vdi \ type=3Dsystem \ virtual-size=3D11GiB ) # Pull the built image into a VDI (Virtual Disk Image) # This operatoin will take a while. (like an hour and a half) xe vdi-import \ uuid=3D${VDIUUID} \ filename=3D/mnt/${FSIMAGE} # Create a VM from the "Other install media" template # Then customize to our taste export TPLUUID=3D$(xe template-list \ name-label=3D"Other install media" \ params=3Duuid --minimal) # Create the VM itself export VM_UUID=3D$(xe vm-install \ new-name-label=3D${VMNAME} \ template=3D${TPLUUID}) # Customize # Defaults: # Desc. Value VM Parameter(s) # RAM =3D 256 MB =3D memory-static-max, memory-dynamic-max, memory-dynamic= -min # 128 MB =3D memory-static-min # CPUs =3D 1 =3D VCPUs-at-startup, VCPUs-max xe vm-param-set uuid=3D${VM_UUID} name-description=3D"New FreeBSD host" xe vm-param-clear uuid=3D${VM_UUID} param-name=3DHVM-boot-policy xe vm-param-clear uuid=3D${VM_UUID} param-name=3DHVM-boot-params xe vm-param-set uuid=3D${VM_UUID} PV-bootloader=3Dpygrub xe vm-param-set uuid=3D${VM_UUID} PV-bootloader-args=3D"--kernel=3D/boot/= kernel/kernel" xe vm-param-set uuid=3D${VM_UUID} other-config:disable_pv_vnc=3D1 ## xe vm-param-set uuid=3D${VM_UUID} other-config:mac_seed=3D'' # Create the Network Interface SWITCH=3D"xenbr0" #=20 # The vendor prefix 00:16:3e is assigned to Xen in oid.txt MAC=3D"00:16:3e:12:34:56" # Edit this value, each VIF must be different ## MAC=3D"random" # if you prefer randomly assigned addresses # Get an ID for the virtual switch. =20 # It is assumed that this is on the first bridge. export NETUUID=3D$(xe network-list bridge=3D${SWITCH} --minimal) export VIFUUID=3D$(xe vif-create \ vm-uuid=3D${VM_UUID} \ network-uuid=3D${NETUUID} \ mac=3D${MAC} \ device=3D0) # Create the Virtual Block Device that links the # VDI to the VM export VBDUUID=3D$(xe vbd-create \ vm-uuid=3D${VM_UUID} \ device=3D0 \ vdi-uuid=3D${VDIUUID} \ type=3DDisk \ mode=3DRW \ bootable=3Dtrue ) =3D=3D=3DEND=3D=3D=3D * Booting it up =3D=3DBEGIN=3D=3D # Starting the first time takes a while xe vm-start uuid=3D${VM_UUID} DOMID=3D$(xe vm-list uuid=3D${VM_UUID} params=3Ddom-id --minimal) /usr/lib/xen/bin/xenconsole ${DOMID} =3D=3D=3DEND=3D=3D=3D # The main difference on the XCP side is that the disk image is bigger. The other steps are the same. I threw in a condenced line to start the VM and fork the console just because it's easier to paste into a terminal window. =3D=3DBEGIN-Advanced-Option=3D=3D VDIUUID=3D$(xe vdi-create \ sr-uuid=3D${SR_UUID} \ name-label=3D${VMNAME}-vdi \ type=3Dsystem \ virtual-size=3D21474836480 ) xe vm-start uuid=3D${VM_UUID} && \ /usr/lib/xen/bin/xenconsole $(xe vm-list uuid=3D${VM_UUID} params=3Ddom= -id --minimal) =3D=3D=3DEND-Advanced-Option=3D=3D=3D * Open Issues I'm probably missing something really obvious here, but here are a couple of issues that keep this from being a complete win. * Kernel ignoring/not-getting arguments from the domain builder. This means that the kernel can''t find its root device. * The pygrub loader won''t execute /boot/loader, so the kernel has to be called directly. No access to loader.conf(5) and the customization it affords. --=20 John D. "Trix" Farrar __\\|//__ Basement.NET trix@basement.net (` o-o ') http://www.basement.net/ -----------------------------------ooO-(_)-Ooo-------------------------- GPG Key Fprint: 525F DBA7 1A62 E4C4 E642 DF95 384B B851 3CEF C10A --WYTEVAkct0FjGQmd Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="pygrub.xcp-ufs-debug.patch" Content-Transfer-Encoding: quoted-printable --- pygrub.xcp-orig 2012-02-09 10:39:29.000000000 -0600 +++ pygrub.xcp-ufs-debug 2012-02-09 10:41:44.000000000 -0600 @@ -12,8 +12,13 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. # +# 2012-02-08 - John D. "Trix" Farrar - 0.6.1-bbkt +# Using XCF version of pygrub, incorporated patches from: +# http://people.freebsd.org/~adrian/xen/bsd_pygrub/ +# which seem to be based on an older version of "0.6" +# Also included a patch to add a --debug switch for easier troubleshootin= g. =20 -import os, sys, atexit, string, struct, tempfile, re +import os, sys, atexit, string, struct, tempfile, re, traceback import copy import logging import platform @@ -26,7 +31,7 @@ import grub.LiloConf import grub.ExtLinuxConf =20 -PYGRUB_VER =3D 0.6 +PYGRUB_VER =3D "0.6.1bbkt" =20 def enable_cursor(ison): if ison: @@ -74,6 +79,105 @@ =20 raise RuntimeError, "No root slice found" =20 =20 +# +# Find the root slice in a FreeBSD disklabel. +# + +# Some magic constants for the BSD disklabel +BSD_LABEL_DISKMAGIC=3D0x82564557 +BSD_LABEL_LABELSECTOR=3D1 +BSD_LABEL_LABELOFFSET=3D0 +BSD_LABEL_MAXPARTITIONS=3D8 +BSD_LABEL_BBSIZE=3D8192 +BSD_LABEL_LABEL_PART=3D2 +BSD_LABEL_RAW_PART=3D2 + +# Magic offsets in the label +BSD_LABEL_OFFSET_MAGIC=3D0 # u_int32_t +BSD_LABEL_OFFSET_SECTORSIZE=3D40 # u_int32_t - number of = bytes per sector +BSD_LABEL_OFFSET_PARTITIONTABLE=3D148 # structure * MAXPARTITI= ONS +BSD_LABEL_OFFSET_NPARTITIONS=3D138 # u_int16_t + +# Magic offsets in the -partition- section +BSD_LABEL_PARTITION_SIZE=3D0 # u_int32_t - size in se= ctors +BSD_LABEL_PARTITION_OFFSET=3D4 # u_int32_t - offset in = sectors +BSD_LABEL_PARTITION_FSTYPE=3D12 # u_int8_t - partition t= ype + +# How big is the partition structure? +BSD_LABEL_PARTITION_STRUCT_SIZE=3D16 + +# BSD label partition types that we care about +BSD_FS_BSDFFS=3D7 + +# Get the partition size from the given freebsd disklabel +# partbuf contains the beginning of the disklabel +def get_freebsd_partition_size(partbuf, partid): + i =3D BSD_LABEL_OFFSET_PARTITIONTABLE + (partid * BSD_LABEL_PARTITION_= STRUCT_SIZE) + return struct.unpack("<I", partbuf[i+BSD_LABEL_PARTITION_SIZE:i+BSD_LA= BEL_PARTITION_SIZE+4])[0] + +def get_freebsd_partition_offset(partbuf, partid): + i =3D BSD_LABEL_OFFSET_PARTITIONTABLE + (partid * BSD_LABEL_PARTITION_= STRUCT_SIZE) + return struct.unpack("<I", partbuf[i+BSD_LABEL_PARTITION_OFFSET:i+BSD_= LABEL_PARTITION_OFFSET+4])[0] + +def get_freebsd_partition_fstype(partbuf, partid): + i =3D BSD_LABEL_OFFSET_PARTITIONTABLE + (partid * BSD_LABEL_PARTITION_= STRUCT_SIZE) + return struct.unpack("<B", partbuf[i+BSD_LABEL_PARTITION_FSTYPE:i+BSD_= LABEL_PARTITION_FSTYPE+1])[0] + +# The FreeBSD disklabel can either be dangerously dedicated or inside a DOS +# partition. This function takes "offset" to mean the beginning of the Fre= eBSD +# disk label and searches for a root partition "a" in the given slice. +# The slice must have the FS type set to something that we know about - th= is +# is currently "BSDFFS" (type 7). Anything else will fail. +# +# The filesystem must be at the beginning of that returned offset for the = FS library +# to discover and handle. + +# The returned offset is relative to the passed in offset - ie the beginni= ng of +# the "disklabel". This is so this code can be used regardless of the unde= rlying +# partitioning scheme (BSD FFS direct on disk, BSD FFS in a DOS primary pa= rtition, +# etc.) + +def get_freebsd_slice(file, offset): + """Find the root slice in a FreeBSD disklabel.""" + print >>sys.stderr, "get_freebsd_slice: file : %s" % file + print >>sys.stderr, "get_freebsd_slice: offset: %s" % offset + + fd =3D os.open(file, os.O_RDONLY) + os.lseek(fd, offset + (SECTOR_SIZE * BSD_LABEL_LABELSECTOR) + BSD_LABE= L_LABELOFFSET, 0) + buf =3D os.read(fd, 1024) + + # Check the magic - is it a valid disklabel? if not, barf + bsd_d_magic =3D struct.unpack("<I", buf[BSD_LABEL_OFFSET_MAGIC:BSD_LAB= EL_OFFSET_MAGIC+4])[0] + print >>sys.stderr, "bsd magic: %s" % bsd_d_magic + print >>sys.stderr, "needed magic: %s" % BSD_LABEL_DISKMAGIC + if bsd_d_magic <> BSD_LABEL_DISKMAGIC: + raise RuntimeError, "FreeBSD label diskmagic incorrect" + + # Get sectorsize - does it make sense? If not, barf + bsd_d_secsize =3D struct.unpack("<I", buf[BSD_LABEL_OFFSET_SECTORSIZE:= BSD_LABEL_OFFSET_SECTORSIZE+4])[0]; + print >>sys.stderr, "bsd sector size: %s" % bsd_d_secsize + + # Get the first partition - is it the right type? If not, barf + bsd_d_npartitions =3D struct.unpack("<H", buf[BSD_LABEL_OFFSET_NPARTIT= IONS:BSD_LABEL_OFFSET_NPARTITIONS+2])[0]; + print >>sys.stderr, "bsd number of partitions: %s" % bsd_d_npartitions + + # Grab the filesystem offset in sectors + bsd_d_part_offset =3D get_freebsd_partition_offset(buf, 0) + bsd_d_part_size =3D get_freebsd_partition_size(buf, 0) + bsd_d_part_fstype =3D get_freebsd_partition_fstype(buf, 0) + print >>sys.stderr, "bsd partition 0 offset: %s" % bsd_d_part_offset + print >>sys.stderr, "bsd partition 0 size : %s" % bsd_d_part_size + print >>sys.stderr, "bsd partition 0 type : %s" % bsd_d_part_fstype + + if bsd_d_part_fstype <> BSD_FS_BSDFFS: + raise RuntimeError, "First BSD partition is not type BSDFFS (7)" + + # XXX one should now verify that the given partition size and offset f= all within the disk size.. + + # Calculate the filesystem offset, return that + return (bsd_d_part_offset * bsd_d_secsize) + + def get_fs_offset_gpt(file): fd =3D os.open(file, os.O_RDONLY) # assume the first partition is an EFI system partition. @@ -81,9 +185,11 @@ buf =3D os.read(fd, 512) return struct.unpack("<Q", buf[32:40])[0] * SECTOR_SIZE =20 +# Partition Types FDISK_PART_SOLARIS=3D0xbf FDISK_PART_SOLARIS_OLD=3D0x82 FDISK_PART_GPT=3D0xee +FDISK_PART_FREEBSD=3D0xa5 =20 def get_partition_offsets(file): if not is_disk_image(file): @@ -112,6 +218,13 @@ except RuntimeError: continue # no solaris magic at that offset, ignore partiti= on =20 + if type =3D=3D FDISK_PART_FREEBSD: + try: + offset +=3D get_freebsd_slice(file, offset) + except RuntimeError: + print >>sys.stderr, "BAIL! I don't know what to do!" + sys.exit(1) + if type =3D=3D FDISK_PART_GPT: offset =3D get_fs_offset_gpt(file) =20 @@ -620,7 +733,7 @@ ["quiet", "interactive", "not-really",= =20 "help", "output=3D", "entry=3D", "kern= el=3D",=20 "ramdisk=3D", "args=3D", "default_args= =3D",=20 - "extra_args=3D", "vm=3D"]) + "extra_args=3D", "vm=3D", "debug"]) except getopt.GetoptError: usage() sys.exit(1) @@ -633,6 +746,7 @@ output =3D None entry =3D None interactive =3D True + debug =3D False not_really =3D False default_args =3D "" extra_args =3D "" @@ -654,6 +768,8 @@ sys.exit() elif o in ("-n", "--not-really"): not_really =3D True + elif o in ("--debug"): + debug =3D True elif o in ("--output",): output =3D a elif o in ("--kernel",): @@ -671,6 +787,9 @@ elif o in ("--extra_args",): extra_args =3D a =20 + if debug: + logging.basicConfig(level=3Dlogging.DEBUG) + if output is None or output =3D=3D "-": fd =3D sys.stdout.fileno() else: @@ -700,6 +819,11 @@ except: # IOErrors raised by fsimage.open # RuntimeErrors raised by run_grub if no menu.lst present + if debug: + traceback.print_exc()=20 + print >>sys.stderr, "Filename: %s" % file + print >>sys.stderr, "Offset : %s" % offset + print >>sys.stderr, "Options : %s" % bootfsoptions fs =3D None continue else: @@ -738,6 +862,8 @@ except: # IOErrors raised by fsimage.open # RuntimeErrors raised by run_grub if no menu.lst present + if debug: + traceback.print_exc()=20 fs =3D None continue =20 --WYTEVAkct0FjGQmd Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="pygrub.xcp-ufs-debug.patch.sig" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEABECAAYFAk80D8oACgkQOEu4UTzvwQpIfACbBF4P6E+3g1ESHv182EquFbyi Dn4An1MTbrOynH9ui85xCk31PukepmSS =XwGn -----END PGP SIGNATURE----- --WYTEVAkct0FjGQmd-- --O3RTKUHj+75w1tg5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk82necACgkQOEu4UTzvwQo1mQCghecWgvlYAYkwBanZD0yrXT/3 IZgAnj1l65FAEROUPL+REiBvLxLbVGD0 =UldF -----END PGP SIGNATURE----- --O3RTKUHj+75w1tg5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120211165711.GB71096>