Date: Sun, 6 May 2018 02:54:03 +0900 From: <iwama@t3.rim.or.jp> To: "'Kurt Lidl'" <lidl@pix.net>, <freebsd-sparc64@freebsd.org> Subject: RE: Root on ZFS RAIDZ1 Message-ID: <00b201d3e49a$0e6e9120$2b4bb360$@t3.rim.or.jp> In-Reply-To: <7b0229eb-1deb-888a-db7c-b973c39251dc@pix.net> References: <000501d3e387$727581b0$57608510$@t3.rim.or.jp> <f4e7d6f2-8040-054a-d8f4-c85e9176c600@pix.net> <000101d3e3ec$2e98cff0$8bca6fd0$@t3.rim.or.jp> <7b0229eb-1deb-888a-db7c-b973c39251dc@pix.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Kurt,
I understood your configuration. It is very reasonable. I am satisfied =
with=20
the mirror configuration as well.
I tried installing again with RAIDZ1 in the procedure you sent to me.=20
Unfortunately, the result was the same as before. That is, as follows:
da0 da1 da2 da3
11.1-RELEASE NG NG SLOW SLOW
NG: not boot, SLOW: boot at very slow
On other, eg amd64 platform, RAIDZ1 will work fine, so is there a =
problem=20
with the sparc64 boot code?
---
Yoshihiko Iwama
-----Original Message-----
From: Kurt Lidl <lidl@pix.net>=20
Sent: Saturday, May 5, 2018 6:18 AM
To: iwama@t3.rim.or.jp; freebsd-sparc64@freebsd.org
Subject: Re: Root on ZFS RAIDZ1
I have my V240 configured to have one set of mirrored drives on the =
first two disks, and a second set of mirrored disks on the last two =
disks.
This allows me to have two completely independent OS installs on the =
machine, and easily switch between them, depending on which disk I boot =
the system from. I use different ZFS pool names on the different =
mirrors, so I can, if needed, import the ZFS pool from the other set of =
disks. Handy for fixing up certain classes of experimental errors on =
the machine.
For a while, I ran with one set of mirrored disks, and just had =
non-mirrored zpools set up on the other two disks, one mounted as =
/usr/src and the other mounted as /usr/obj. That experiment didn't show =
any huge benefit, so I just went to my current setup, with two sets of =
independent mirrored disks.
I never tried a RAIDZ of all four disks on the sparc64.
-Kurt
On 5/4/18 5:09 PM, iwama@t3.rim.or.jp wrote:
> Hi Kurt,
>=20
> Thank you for your reply.
>=20
> Thank you for providing your achievements and procedures. I will=20
> try it in your procedure.
>=20
> Please let me ask a question.
> In your procedure, the RAID method is mirror, but is there a record =
> of RAIDZ of 4 disks with V240?
>=20
> Regards,
> ---
> Yoshihiko Iwama
>=20
> -----Original Message-----
> From: owner-freebsd-sparc64@freebsd.org=20
> <owner-freebsd-sparc64@freebsd.org>
> On Behalf Of Kurt Lidl
> Sent: Friday, May 4, 2018 10:31 PM
> To: freebsd-sparc64@freebsd.org
> Subject: Re: Root on ZFS RAIDZ1
>=20
> I have been using various sparc64 machines with a ZFS only=20
> installation for quite a while, and found that it works reliably for=20
> me. My earlier machines were Netra-T1 (all retired now), and=20
> currently I have both a V240 and three
> V120 machines using this configuration.
>=20
> Here are my notes that I use during the partitioning phase of the =
install.
> (ie, start a normal install as discussed in the web page you mention,=20
> and during the 'Shell' method of partitioning, I run these commands):
>=20
> You may or may not want to execute the commented out sysctl, which=20
> will force the ZFS installation to use 4096 byte blocks.
> Given that all the disks I've ever used on sparc64 are really 512byte=20
> block devices, I'm not sure if this is a great idea or not. I=20
> typically do run with 4096 byte blocks.
>=20
> -----------------------------------------------------------------
>=20
> kldload zfs
>=20
> # force 4K minimum zfs block size
> #sysctl vfs.zfs.min_auto_ashift=3D12
>=20
> # appropriate for a 36GB disk w/4+GB of swap zfssize=3D30000M #=20
> appropriate for a 72GB disk w/4+GB of swap zfssize=3D65500M #=20
> appropriate for a 72GB disk w/8+GB of swap zfssize=3D61500M
>=20
> disks=3D"da0 da1"
> for d in $disks
> do
> gpart destroy -F $d
> gpart create -s VTOC8 $d
> gpart add -t freebsd-zfs -s $zfssize $d
> gpart add -t freebsd-swap $d
> gpart bootcode -p /boot/zfsboot $d
> dd if=3D/boot/zfsloader of=3D/dev/${d}a bs=3D512 =
oseek=3D1024 \
> conv=3Dnotrunc,sync
> done
>=20
> # note, we use -R /mnt to set a different root mountpoint P=3Dzroot=20
> mnt=3D/mnt zpool create -f -o cachefile=3Dnone -O checksum=3Dfletcher4 =
\
> -R $mnt -m none $P mirror /dev/da0a /dev/da1a
>=20
> zfs create $P/ROOT
> zfs create $P/ROOT/default
> zpool set bootfs=3D$P/ROOT/default $P
>=20
> # because of "-R /mnt" earlier, this is "/mnt" effectively for this=20
> session zfs set mountpoint=3D/ $P/ROOT/default
>=20
> zfs create $P/vartmp
> zfs create $P/usrports
> zfs create $P/usrsrc
> zfs create $P/usrobj
> zfs create $P/home
> zfs set mountpoint=3D/var/tmp $P/vartmp
> zfs set mountpoint=3D/home $P/home
> zfs set mountpoint=3D/usr/ports $P/usrports zfs set =
mountpoint=3D/usr/src=20
> $P/usrsrc zfs set mountpoint=3D/usr/obj $P/usrobj zfs mount -a
>=20
> gmirror label -b prefer gswap0 da0b da1b
>=20
> eeprom boot-device=3D"disk0 disk1"
>=20
> exit
>=20
> -----------------------------------------------------------------
>=20
> At the end of the installation process, you will need to fix up the=20
> /etc/rc.conf and /boot/loader.conf too:
>=20
> # Add to /etc/rc.conf:
> clear_tmp_enable=3D"YES"
> ntpdate_enable=3D"YES"
> rwhod_enable=3D"YES"
> rwhod_flags=3D"-m"
> zfs_enable=3D"YES"
>=20
> # Add to /boot/loader.conf:
> zfs_load=3D"YES"
> geom_mirror_load=3D"YES"
> vfs.zfs.prefetch_disable=3D1
>=20
> # Add to /etc/fstab:
> # Device Mountpoint FStype Options Dmp Pass#
> /dev/mirror/gswap none swap sw 0 =
0
> #md /tmp mfs rw,-s64m 0 =
0
> tmpfs /tmp tmpfs =
rw,size=3D64m,mode=3D1777 0 0
>=20
> # turn off ttyv? entries in /etc/ttys
> # turn off ttyu2 entry in /etc/ttys
>=20
> I hope this helps you get the system working well.
>=20
> Best wishes,
>=20
> -Kurt
>=20
> On 5/4/18 5:08 AM, iwama@t3.rim.or.jp wrote:
>> I want Sun Fire V245 to use FreeBSD-11.1-RELEASE with Root on ZFS
> RAIDZ1.
>> The installation procedure of Root on ZFS is as follow page:
>>
>> https://wiki.freebsd.org/RootOnZFS/VTOC8ZFSBoot/9.x-RELEASE
>>
>> I finished the installation without any problem, but it was some=20
>> problems at reboot.
>>
>> 1) It do not boot at all
>> 2) boot takes a very long time, more than 3 HOURS
>> 3) It takes some time for boot, about 10 minutes
>>
>> I tried RAIDZ1 with 9.3-RELEASE, RAIDZ2, RAIDZ10 and mirror with=20
>> 11.1-RELEASE, the results were as follows, da0-3 are the name of boot =
>> disk. (Fire V245 has four disks):
>>
>> RAIDZ1(zpool create -o altroot=3D/mnt -O canmount=3Doff zroot raidz1 =
da0a=20
>> da1a da2a da3a)
>> da0 da1 da2 da3
>> 11.1-RELEASE NG NG SLOW SLOW
>> 9.3-RELEASE NG NG SLOW SLOW
>>
>> RAIDZ2(zpool create -o altroot=3D/mnt -O canmount=3Doff zroot raidz2 =
da0a=20
>> da1a da2a da3a)
>> da0 da1 da2 da3
>> 11.1-RELEASE SLOW SLOW SLOW SLOW
>>
>> RAIDZ10(zpool create -o altroot=3D/mnt -O canmount=3Doff zroot mirror =
>> da0a da1a mirror da2a da3a)
>> da0 da1 da2 da3
>> 11.1-RELEASE slow slow slow slow
>>
>> mirror(zpool create -o altroot=3D/mnt -O canmount=3Doff zroot mirror =
da0a=20
>> da1a da2a da3a)
>> da0 da1 da2 da3
>> 11.1-RELEASE OK OK OK OK
>>
>> OK: OK(boot at normal speed), NG: 1), SLOW: 2), slow: 3)
>>
>>
>> From the above results and the logs below, there is no problem=20
>> with mirror, but it is a big problem in RAIDZ1 or RAIDZ2.
>> Has anyone been running Root on RAIDZ1 on the sparc64 platform?
>>
>>
>> Install steps(only the part entered manually):
>> =3D=3D=3D
>> # zpool labelclear -f /dev/da0a
>> # zpool labelclear -f /dev/da1a
>> # zpool labelclear -f /dev/da2a
>> # zpool labelclear -f /dev/da3a
>> #
>> # sysctl kern.disks
>> kern.disks: cd0 da3 da2 da1 da0
>> # gpart destroy -F da0
>> da0 destroyed
>> # gpart destroy -F da1
>> da1 destroyed
>> # gpart destroy -F da2
>> da2 destroyed
>> # gpart destroy -F da3
>> da3 destroyed
>> # gpart create -s VTOC8 da0
>> da0 created
>> # gpart create -s VTOC8 da1
>> da1 created
>> # gpart create -s VTOC8 da2
>> da2 created
>> # gpart create -s VTOC8 da3
>> da3 created
>> #
>> # gpart add -s 270G -t freebsd-zfs da0 da0a added # gpart add -s 270G =
>> -t freebsd-zfs da1 da1a added # gpart add -s 270G -t freebsd-zfs da2=20
>> da2a added # gpart add -s 270G -t freebsd-zfs da3 da3a added # gpart=20
>> add -t freebsd-swap da0 da0b added # gpart add -t freebsd-swap da1=20
>> da1b added # gpart add -t freebsd-swap da2 da2b added # gpart add -t=20
>> freebsd-swap da3 da3b added # ***The following lines differ depending =
>> on the RAID method # zpool create -o altroot=3D/mnt -O canmount=3Doff =
>> zroot raidz1 da0a da1a da2a da3a # # zpool export zroot # gpart=20
>> bootcode -p /boot/zfsboot da0 # gpart bootcode -p /boot/zfsboot da1 # =
>> gpart bootcode -p /boot/zfsboot da2 # gpart bootcode -p /boot/zfsboot
>> da3 # sysctl kern.geom.debugflags=3D0x10
>> kern.geom.debugflags: 0 -> 16
>> # dd if=3D/boot/zfsloader of=3D/dev/da0a bs=3D512 oseek=3D1024=20
>> conv=3Dnotrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.845066 secs (84042 bytes/sec) # dd=20
>> if=3D/boot/zfsloader of=3D/dev/da1a bs=3D512 oseek=3D1024 =
conv=3Dnotrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.832603 secs (84411 bytes/sec) # dd=20
>> if=3D/boot/zfsloader of=3D/dev/da2a bs=3D512 oseek=3D1024 =
conv=3Dnotrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.815799 secs (84915 bytes/sec) # dd=20
>> if=3D/boot/zfsloader of=3D/dev/da3a bs=3D512 oseek=3D1024 =
conv=3Dnotrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.828102 secs (84546 bytes/sec) # zpool=20
>> import -o altroot=3D/mnt zroot #
>> # zfs set checksum=3Dfletcher4 =
zroot
>> # zfs set atime=3Doff =
zroot
>> #
>> # zfs create -o mountpoint=3D/ =
zroot/ROOT
>>
>> #
>> # zpool set bootfs=3Dzroot/ROOT zroot
>> #
>> # cat << EOF > /tmp/bsdinstall_etc/fstab
>>> # Device Mountpoint FStype Options Dump Pass#
>>> /dev/da0b none swap sw 0 0
>>> /dev/da1b none swap sw 0 0
>>> /dev/da2b none swap sw 0 0
>>> /dev/da3b none swap sw 0 0
>>> ^D#
>> #
>> # exit
>>
>>
>> # echo 'zfs_enable=3D"YES"' >> /etc/rc.conf # # cat /boot/loader.conf =
>> zfs_load=3D"YES"
>> ***Except for 9.3-RELEASE, since the line of "zfs_load=3D"YES"" is=20
>> originally in ***/boot/loader.conf, addition is unnecessary #
>> # zpool set cachefile=3D/boot/zfs/zpool.cache zroot
>> # exit
>>
>> ***In 11.1-RELEASE, the last "zpool set=20
>> cachefile=3D/boot/zfs/zpool.cache zroot" failed,
>> shown as "internal error: failed to initialize ZFS library"
>> =3D=3D=3D
>>
>>
>> NOT boot log:
>> =3D=3D=3D
>> {1} ok boot disk0
>>
>> SC Alert: Host System has Reset
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>> screen not found.
>> keyboard not found.
>> Keyboard not present. Using ttya for input and output.
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>>
>>
>> Sun Fire V245, No Keyboard
>> Copyright 2006 Sun Microsystems, Inc. All rights reserved.
>> OpenBoot 4.22.22, 16384 MB memory installed, Serial #XXXXXXXX.
>> Ethernet address 0:14:4f:XX:XX:XX, Host ID: XXXXXXXX.
>>
>>
>>
>> Initializing 1MB of memory at addr 137ff0a000
>> Initializing 1MB of memory at addr 137ff06000
>> Initializing 1MB of memory at addr 137ff00000
>> Initializing 1MB of memory at addr 137fee0000
>> Initializing 1MB of memory at addr 137fe90000
>> Initializing 1MB of memory at addr 137fdea000
>> Initializing 14MB of memory at addr 137f000000
>> Initializing 2032MB of memory at addr 1300000000
>> Initializing 2048MB of memory at addr 1200000000
>> Initializing 2048MB of memory at addr 1100000000
>> Initializing 2048MB of memory at addr 1000000000
>> Initializing 2048MB of memory at addr 300000000
>> Initializing 2048MB of memory at addr 200000000
>> Initializing 2048MB of memory at addr 100000000
>> Initializing 2048MB of memory at addr 0
>> Rebooting with command: boot disk0
>> Boot device: /pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/disk@0,0
>> File and args:
>> =20
>>>> FreeBSD/sparc64 ZFS boot block
>> Boot path: =
/pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/disk@0,0:a
>> Consoles: Open Firmware console
>> ZFS: i/o error - all block copies unavailable
>> ZFS: can't read MOS object directory
>> ZFS: can't find root filesystem
>>
>> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0 (Fri Jul=20
>> 21
>> 08:44:42 UTC 2017 root@releng2.nyi.freebsd.org) bootpath=3D""
>>
>> can't load 'kernel'
>>
>> Type '?' for a list of commands, 'help' for more detailed help.
>> OK reboot
>> Rebooting...
>> Program terminated
>> {1} ok
>> =3D=3D=3D
>>
>>
>> VERY SLOW boot log:
>> =3D=3D=3D
>> {1} ok boot disk3
>>
>> SC Alert: Host System has Reset
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>> screen not found.
>> keyboard not found.
>> Keyboard not present. Using ttya for input and output.
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>>
>>
>> Sun Fire V245, No Keyboard
>> Copyright 2006 Sun Microsystems, Inc. All rights reserved.
>> OpenBoot 4.22.22, 16384 MB memory installed, Serial #XXXXXXXX.
>> Ethernet address 0:14:4f:XX:XX:XX, Host ID: XXXXXXXX.
>>
>>
>>
>> Initializing 1MB of memory at addr 137ff0a000
>> Initializing 1MB of memory at addr 137ff06000
>> Initializing 1MB of memory at addr 137ff00000
>> Initializing 1MB of memory at addr 137fee0000
>> Initializing 1MB of memory at addr 137fe90000
>> Initializing 1MB of memory at addr 137fdea000
>> Initializing 14MB of memory at addr 137f000000
>> Initializing 2032MB of memory at addr 1300000000
>> Initializing 2048MB of memory at addr 1200000000
>> Initializing 2048MB of memory at addr 1100000000
>> Initializing 2048MB of memory at addr 1000000000
>> Initializing 2048MB of memory at addr 300000000
>> Initializing 2048MB of memory at addr 200000000
>> Initializing 2048MB of memory at addr 100000000
>> Initializing 2048MB of memory at addr 0
>> Rebooting with command: boot disk3
>> Boot device: /pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/disk@3,0
>> File and args:
>> =20
>>>> FreeBSD/sparc64 ZFS boot block
>> Boot path: =
/pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/disk@3,0:a
>> Consoles: Open Firmware console
>>
>>
>> ***Slow from here(\|/-\|/- animation)***
>> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0 (Fri Jul=20
>> 21
>> 08:44:42 UTC 2017 root@releng2.nyi.freebsd.org)=20
>> bootpath=3D"zfs:zroot/ROOT:"
>> Loading /boot/defaults/loader.conf
>> /boot/kernel/kernel data=3D0xc47740+0xf4078=20
>> syms=3D[0x8+0xd5a38+0x8+0xc746a] /boot/entropy size=3D0x1000=20
>> /boot/kernel/zfs.ko text=3D0x25cc60 data=3D0xb9d0+0x90050
>> |syms=3D[0x8+0x1bd38/-+0x8+0x16db2]
>> loading required module 'opensolaris'
>> /boot/kernel/opensolaris.ko text=3D0x3730 data=3D0x2c8+0x1a30=20
>> syms=3D[0x8+0xe10+0x8+0x968] ***It is slow until the above is
>> displayed.***
>>
>>
>> Hit [Enter] to boot immediately, or any other key for command prompt.
>> Booting [/boot/kernel/kernel] in 9 seconds... Booting=20
>> [/boot/kernel/kernel] in 8 seconds... Booting [/boot/kernel/kernel]=20
>> in
>> 7 seconds... Booting [/boot/kernel/kernel] in 6 seconds... Booting=20
>> [/boot/kernel/kernel] in 5 seconds... Booting [/boot/kernel/kernel]=20
>> in
>> 4 seconds... Booting [/boot/kernel/kernel] in 3 seconds... Booting=20
>> [/boot/kernel/kernel] in 2 seconds... Booting [/boot/kernel/kernel]=20
>> in
>> 1 second... Booting [/boot/kernel/kernel]...
>> -\|/-\|/-\|/-\|/-\|jumping to kernel entry at 0xc00b0000.
>> =3D=3D=3D
>>
>> ---
>> Yoshihiko Iwama
>>
>>
>> _______________________________________________
>> freebsd-sparc64@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
>> To unsubscribe, send any mail to =
"freebsd-sparc64-unsubscribe@freebsd.org"
>>
>=20
> _______________________________________________
> freebsd-sparc64@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
> To unsubscribe, send any mail to =
"freebsd-sparc64-unsubscribe@freebsd.org"
>=20
> _______________________________________________
> freebsd-sparc64@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
> To unsubscribe, send any mail to =
"freebsd-sparc64-unsubscribe@freebsd.org"
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?00b201d3e49a$0e6e9120$2b4bb360$>