Date: Thu, 9 Jul 2020 14:18:54 -0700 From: John Kennedy <warlock@phouka.net> To: Kyle Evans <kevans@freebsd.org> Cc: Guido van Rooij <guido@gvr.org>, FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: 12.1p7 no longer boots after doing zpool upgrade -a Message-ID: <20200709211854.GA11731@phouka1.phouka.net> In-Reply-To: <CACNAnaGbCknS18yLv1ow2FFnj5xSMHXQZgxfRPMUMTb5ujB=fw@mail.gmail.com> References: <20200709131201.GA3464@co.gvr.org> <CACNAnaGbCknS18yLv1ow2FFnj5xSMHXQZgxfRPMUMTb5ujB=fw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jul 09, 2020 at 08:24:54AM -0500, Kyle Evans wrote: > On Thu, Jul 9, 2020 at 8:12 AM Guido van Rooij <guido@gvr.org> wrote: > > > > I did a zpool upgrade -a to enable large_dnode and spacemap_v2. > > After that, I did: > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0 > > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1 > > and: > > gpart bootcode -p /boot/boot1.efifat -i 1 ada0 > > gpart bootcode -p /boot/boot1.efifat -i 1 ada1 > > > > Now the system no longer boots from either disk and drops to the efi shell. > > This method of updating the ESP is no longer recommended for new 12.x > installations -- we now more carefully construct the ESP with an > /EFI/FreeBSD/loader.efi where loader.efi is /boot/loader.efi. You will > want to rebuild this as such, and that may fix part of your problem. So, I've got a stable box: FreeBSD 12.1-STABLE #156 r362983+cd7a9efd329c(stable/12): Tue Jul 7 07:42:17 PDT 2020 I've got almost the same situation, except I haven't shot myself in the foot yet. My partitions: # gpart show nvd0 nvd1 => 40 976773088 nvd0 GPT (466G) 40 409600 1 efi (200M) 409640 1024 2 freebsd-boot (512K) 410664 984 - free - (492K) 411648 16777216 3 freebsd-swap (8.0G) 17188864 959584256 4 freebsd-zfs (458G) 976773120 8 - free - (4.0K) => 40 976773088 nvd1 GPT (466G) 40 409600 1 efi (200M) 409640 1024 2 freebsd-boot (512K) 410664 984 - free - (492K) 411648 16777216 3 freebsd-swap (8.0G) 17188864 959584256 4 freebsd-zfs (458G) 976773120 8 - free - (4.0K) So this is my first UEFI system that's hit an upgrade point: # zpool status pool: zroot state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0 in 0 days 00:02:04 with 0 errors on Wed Jul 1 13:21:57 2020 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 nvd1p4.eli ONLINE 0 0 0 nvd0p4.eli ONLINE 0 0 0 I usually seed these things off of stick and they evolve from there, but this one is slightly interesting because the motherboard died repeatedly and I'm guessing that the M.2 drives got reinstalled in the "wrong" order so now if I don't manually select one of the M.2 drives, it won't reboot (acts like the encryption password is bad, fails out into a shell where I reboot). In any case, I know if I upgrade, it is going to tell me to upgrade the boot code and I'm wondering what that is these days. If this was a Pre-UEFI, I would be expecting to do (only) something like: gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nvd1 IE, might hit either drive, update the freebsd-boot partition by # on both, and special note that the # is now one higher because of the EFI partition. The FreeBSD wiki still has the "old" way of doing it here... https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot ... but your newer way of doing it here: https://wiki.freebsd.org/UEFI So the newer way of doing it would be something like this, except I'd by updating the file rather than looking at it's checksum/version? strings -a < /boot/loader.efi | grep -A1 'EFI loader' FreeBSD/amd64 EFI loader, Revision 1.1 (Tue Jul 7 07:33:24 PDT 2020 warlock@ouroboros.phouka.net) mount -t msdosfs /dev/nvd0p1 /mnt md5 -r /boot/loader.efi /mnt/efi/boot/BOOTx64.efi 9f64a96f3170e3327ca7dad1d574d852 /boot/loader.efi cbff58cc8ee79ce7342f5be23ad14ba3 /mnt/efi/boot/BOOTx64.efi strings -a < /mnt/efi/boot/BOOTx64.efi | grep -A1 'EFI loader' FreeBSD/amd64 EFI loader, Revision 1.1 |/-\ umount -v /mnt mount -t msdosfs /dev/nvd1p1 /mnt md5 -r /mnt/efi/boot/BOOTx64.efi cbff58cc8ee79ce7342f5be23ad14ba3 /mnt/efi/boot/BOOTx64.efi umount -v /mnt At least some of the checksum difference is because WITHOUT_REPRODUCIBLE_BUILD=YES. (Sorry for extra details, but I know when I was researching this, I ran across a *lot* of examples where we didn't have a GPT dump and people were just using partition #s and if you stomp on the wrong partition you're going to have a bad day, so leaving some specific examples here.) So one recipe doesn't even seem to make the freebsd-boot partition, so is it optional for a pure UEFI boot? Should we always gpart-bootcode it if it exists and upgrade BOOTx64.efi when the EFI partition exists, or is there a few more wrinkles we need to worry about? In any case, is it a logical theory that my possible boot-order problem is because drive order got swapped and maybe one wasn't properly updated? They seem to be the same: # md5 /dev/nvd[01]p2 MD5 (/dev/nvd0p2) = 2ded438a2c97ea551446cc2d1d3b498e MD5 (/dev/nvd1p2) = 2ded438a2c97ea551446cc2d1d3b498e Ideally I'd like to have not boot through the UEFI boot menu every time. I'm not sure why the drive order seems to matter right now.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200709211854.GA11731>