Date: Sun, 18 May 2014 03:53:26 -0700 From: Jeff Chan <jeffc@supranet.net> To: freebsd-fs@freebsd.org Subject: Re: ZFS snapshot restore not quite working; missing steps? Message-ID: <236659679.20140518035326@supranet.net> In-Reply-To: <F8DBD16F6101498A98AA04D29C1FD596@multiplay.co.uk> References: <1129127016.20140517185154@supranet.net> <1198503903.20140518030907@supranet.net> <F8DBD16F6101498A98AA04D29C1FD596@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sunday, May 18, 2014, 3:25:39 AM, Steven Hartland wrote: > Not used legacy here have you tried: > zfs set mountpoint=3D/ zroot It was tried initially. Trying again a second time zfs set mountpoint=3Dlegacy zroot was tried. > Silly question given your using mfi have confirmed your controller / > machine bios are set to boot from the relavent disk? Not silly; yes it does boot from the right disk. > Also have you confirmed zfs in /boot/loader.conf e.g. > zfs_load=3D"YES" > vfs.root.mountfrom=3D"zfs:zroot" Yes. > Finally do be aware its not ideal to use a RAID controller for ZFS > your better of with a HBA which does less "fancy" stuff when > communicating with the disks allowing ZFS do see whats really going > on. Yes, indeed. It's an LSI 2208 which doesn't seem to be reflashable into HBA like a LSI 2008, so we're running each of the three disks as a separate RAID-0, with RAID hardware read and write caching turned off (as it supposedly works faster with ZFS that way). Would have preferred HBA, but we're trying to do the best with the hardware we have.=20 We wonder if things like the MBR and zfs boot loader are happy. How does one review whether those are set up correctly? Cheers, Jeff C. > Regards > Steve > ----- Original Message -----=20 > From: "Jeff Chan" <jeffc@supranet.net> > To: <freebsd-fs@freebsd.org> > Sent: Sunday, May 18, 2014 11:09 AM > Subject: Re: ZFS snapshot restore not quite working; missing steps? >> On Saturday, May 17, 2014, 6:51:54 PM, Jeff Chan wrote: >>> We're trying to do a ZFS snapshot restore of a ZFS non-RAIDZ* FreeBSD >>> 9.2-RELEASE to a new bare system configured with ZFS RAIDZ, and it's not >>> quite working right. The restore itself seems to complete, but we're >>> not able to successfully boot the resulting system. We can't recall >>> the exact error we got, but think the ZFS cache file was not found. >>=20 >>> Presumably we're missing some steps, don't have all magic for ZFS >>> booting from a FreeBSD root partition installed/configured correctly, >>> etc. What did we miss? >>=20 >>> (We have a private network to serve the snapshot over NFS from a third >>> server on 192.168.0.2.) >>=20 >>=20 >>> (There may be some line wrap below.) >>=20 >>> (Comments in (parens)) >>=20 >>> (Some outputs in [brackets]) >>=20 >>=20 >>=20 >>=20 >>> On the old system, call it foo: >>=20 >>=20 >>> foo: [103]% zfs list >>> NAME USED AVAIL REFER MOUNTPOINT >>> zroot 169G 3.40T 545M legacy >>> zroot/home 15.1G 3.40T 10.3G /home >>> zroot/home/username 4.04G 3.40T 2.74G /home/username >>> zroot/bar 128G 3.40T 97K /bar >>> zroot/bar/prod 126G 3.40T 73.8G /bar/prod >>> zroot/bar/test 2.18G 3.40T 2.18G /bar/test >>> zroot/tmp 1.35G 3.40T 1.35G /tmp >>> zroot/usr 9.60G 3.40T 9.52G /usr >>> zroot/var 14.0G 3.40T 14.0G /var >>> foo: [104]% zpool list >>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>> zroot 3.62T 170G 3.46T 4% 1.00x ONLINE - >>=20 >>=20 >>=20 >>> On the new system: >>=20 >>=20 >>> Boot from a FreeBSD 9.2-RELEASE installer USB flash drive >>> into LiveCD, then: >>=20 >>> (bring up NFS private LAN) >>> ifconfig ix3 192.168.0.3 netmask 255.255.255.0 up >>> [ix2: Could not setup receive structures >>> [ix2: Could not setup receive structures >> ->> >>> sysctl kern.ipc.nmbclusters=3D131072 >>> sysctl kern.ipc.nmbjumbo9=3D38400 >>=20 >>> (mount remote NFS backup directory) >>> mount -t nfs 192.168.0.2:/home/backup /mnt >>> mkdir /var/mnt >>=20 >>> sysctl kern.disks >>> [kern.disks: da0 mfid2 mfid1 mfid0] >>=20 >>> gpart create -s gpt mfid0 >>> gpart create -s gpt mfid1 >>> gpart create -s gpt mfid2 >>> gpart add -s 222 -a 4k -t freebsd-boot -l boot0 mfid0 >>> gpart add -s 222 -a 4k -t freebsd-boot -l boot1 mfid1 >>> gpart add -s 222 -a 4k -t freebsd-boot -l boot2 mfid2 >>> gpart add -s 8g -a 4k -t freebsd-swap -l swap0 mfid0 >>> gpart add -s 8g -a 4k -t freebsd-swap -l swap1 mfid1 >>> gpart add -s 8g -a 4k -t freebsd-swap -l swap2 mfid2 >>> gpart add -a 4k -t freebsd-zfs -l disk0 mfid0 >>> gpart add -a 4k -t freebsd-zfs -l disk1 mfid1 >>> gpart add -a 4k -t freebsd-zfs -l disk2 mfid2 >>> (Clear any old zfs data) >>> dd if=3D/dev/zero of=3D/dev/mfid0p3 count=3D560 bs=3D512 >>> dd if=3D/dev/zero of=3D/dev/mfid1p3 count=3D560 bs=3D512 >>> dd if=3D/dev/zero of=3D/dev/mfid2p3 count=3D560 bs=3D512 >>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid0 >>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid1 >>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 mfid2 >>> zpool create -f -m none -o altroot=3D/var/mnt -o >>> cachefile=3D/tmp/zpool.cache zroot raidz mfid0 mfid1 mfid2 >>=20 >>> gunzip -c /mnt/foo-backup-full/zroot@20140516183425.gz | zfs receive -v= dFu zroot >>> zpool set bootfs=3Dzroot zroot >>> zpool get all zroot >>> zfs get all zroot >>=20 >>> zfs set mountpoint=3D/var/mnt zroot >>> (add swap to fstab) >>> vi /var/mnt/var/mnt/etc/fstab >>> [/dev/mfid0p2 none swap sw 0 0 >>> [/dev/mfid1p2 none swap sw 0 0 >>> [/dev/mfid2p2 none swap sw 0 0 >>=20 >>> vi /var/mnt/var/mnt/etc/rc.conf [ change adapter name & IP ] >>> vi /var/mnt/var/mnt/etc/pf.conf [ change adapter name ] >>> [vi /var/mnt/var/mnt/boot/loader.conf] (not performed) >>> cd / >>> zpool export zroot >>> zpool import -o altroot=3D/var/mnt -o cachefile=3D/tmp/zpool.cache zroot >>> [zfs set mountpoint=3D/ zroot] (not performed) >>> cp /tmp/zpool.cache /var/mnt/boot/zfs >>> zpool get all zroot >>> zpool set cachefile=3D/boot/zfs/zpool.cache zroot >>> zfs unmount -a >>> zfs set mountpoint=3Dlegacy zroot >>> zfs set mountpoint=3D/home zroot/home >>> zfs set mountpoint=3D/bar zroot/bar >>> zfs set mountpoint=3D/tmp zroot/tmp >>> zfs set mountpoint=3D/usr zroot/usr >>> zfs set mountpoint=3D/var zroot/var >>=20 >>> [remove USB] >>> reboot >>=20 >>=20 >>=20 >> Additional information, gpart list on the new system: >>=20 >>=20 >> root@:~ # gpart list >> Geom name: da0 >> modified: false >> state: OK >> fwheads: 255 >> fwsectors: 63 >> last: 7821311 >> first: 0 >> entries: 8 >> scheme: BSD >> Providers: >> 1. Name: da0a >> Mediasize: 734167040 (700M) >> Sectorsize: 512 >> Mode: r1w0e1 >> rawtype: 7 >> length: 734167040 >> offset: 0 >> type: freebsd-ufs >> index: 1 >> end: 1433919 >> start: 0 >> Consumers: >> 1. Name: da0 >> Mediasize: 4004511744 (3.7G) >> Sectorsize: 512 >> Mode: r1w0e1 >>=20 >> Geom name: mfid0 >> modified: false >> state: OK >> fwheads: 255 >> fwsectors: 63 >> last: 5859373022 >> first: 34 >> entries: 128 >> scheme: GPT >> Providers: >> 1. Name: mfid0p1 >> Mediasize: 110592 (108k) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 20480 >> Mode: r0w0e0 >> rawuuid: fe8923b4-ddd6-11e3-8d53-a0369f312ea0 >> rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f >> label: boot0 >> length: 110592 >> offset: 20480 >> type: freebsd-boot >> index: 1 >> end: 255 >> start: 40 >> 2. Name: mfid0p2 >> Mediasize: 8589934592 (8.0G) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 131072 >> Mode: r0w0e0 >> rawuuid: 0863e7f4-ddd7-11e3-8d53-a0369f312ea0 >> rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b >> label: swap0 >> length: 8589934592 >> offset: 131072 >> type: freebsd-swap >> index: 2 >> end: 16777471 >> start: 256 >> 3. Name: mfid0p3 >> Mediasize: 2991408918528 (2.7T) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 131072 >> Mode: r0w0e0 >> rawuuid: 08a7dd63-ddd7-11e3-8d53-a0369f312ea0 >> rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b >> label: disk0 >> length: 2991408918528 >> offset: 8590065664 >> type: freebsd-zfs >> index: 3 >> end: 5859373015 >> start: 16777472 >> Consumers: >> 1. Name: mfid0 >> Mediasize: 2999999004672 (2.7T) >> Sectorsize: 512 >> Mode: r0w0e0 >>=20 >> Geom name: mfid1 >> modified: false >> state: OK >> fwheads: 255 >> fwsectors: 63 >> last: 5859373022 >> first: 34 >> entries: 128 >> scheme: GPT >> Providers: >> 1. Name: mfid1p1 >> Mediasize: 110592 (108k) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 20480 >> Mode: r0w0e0 >> rawuuid: 02b8c77a-ddd7-11e3-8d53-a0369f312ea0 >> rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f >> label: boot1 >> length: 110592 >> offset: 20480 >> type: freebsd-boot >> index: 1 >> end: 255 >> start: 40 >> 2. Name: mfid1p2 >> Mediasize: 8589934592 (8.0G) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 131072 >> Mode: r0w0e0 >> rawuuid: 086d2831-ddd7-11e3-8d53-a0369f312ea0 >> rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b >> label: swap1 >> length: 8589934592 >> offset: 131072 >> type: freebsd-swap >> index: 2 >> end: 16777471 >> start: 256 >> 3. Name: mfid1p3 >> Mediasize: 2991408918528 (2.7T) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 131072 >> Mode: r0w0e0 >> rawuuid: 08b854b6-ddd7-11e3-8d53-a0369f312ea0 >> rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b >> label: disk1 >> length: 2991408918528 >> offset: 8590065664 >> type: freebsd-zfs >> index: 3 >> end: 5859373015 >> start: 16777472 >> Consumers: >> 1. Name: mfid1 >> Mediasize: 2999999004672 (2.7T) >> Sectorsize: 512 >> Mode: r0w0e0 >>=20 >> Geom name: mfid2 >> modified: false >> state: OK >> fwheads: 255 >> fwsectors: 63 >> last: 5859373022 >> first: 34 >> entries: 128 >> scheme: GPT >> Providers: >> 1. Name: mfid2p1 >> Mediasize: 110592 (108k) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 20480 >> Mode: r0w0e0 >> rawuuid: 03c0b756-ddd7-11e3-8d53-a0369f312ea0 >> rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f >> label: boot2 >> length: 110592 >> offset: 20480 >> type: freebsd-boot >> index: 1 >> end: 255 >> start: 40 >> 2. Name: mfid2p2 >> Mediasize: 8589934592 (8.0G) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 131072 >> Mode: r0w0e0 >> rawuuid: 087d8c31-ddd7-11e3-8d53-a0369f312ea0 >> rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b >> label: swap2 >> length: 8589934592 >> offset: 131072 >> type: freebsd-swap >> index: 2 >> end: 16777471 >> start: 256 >> 3. Name: mfid2p3 >> Mediasize: 2991408918528 (2.7T) >> Sectorsize: 512 >> Stripesize: 0 >> Stripeoffset: 131072 >> Mode: r0w0e0 >> rawuuid: 08c61aed-ddd7-11e3-8d53-a0369f312ea0 >> rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b >> label: disk2 >> length: 2991408918528 >> offset: 8590065664 >> type: freebsd-zfs >> index: 3 >> end: 5859373015 >> start: 16777472 >> Consumers: >> 1. Name: mfid2 >> Mediasize: 2999999004672 (2.7T) >> Sectorsize: 512 >> Mode: r0w0e0 >>=20 >>=20 >>=20 >>=20 >> -- >> Jeff Chan >> mailto:jeffc@supranet.net >>=20 >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> -- Jeff Chan mailto:jeffc@supranet.net http://www.jeffchan.com/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?236659679.20140518035326>