Date: Mon, 11 Nov 2019 17:10:15 +1100 From: Peter Jeremy <peter@rulingia.com> To: Jonathan Anderson <jonathan.anderson@mun.ca> Cc: "freebsd-fs@FreeBSD.org" <freebsd-fs@freebsd.org> Subject: Re: Broken ZFS boot on upgrade Message-ID: <20191111061015.GA50716@server.rulingia.com> In-Reply-To: <CAP8WKbJWSHzhFCKijRVxydKEwgD_4NX2gmA-QVEVZPuotFCGvQ@mail.gmail.com> References: <CAP8WKbJWSHzhFCKijRVxydKEwgD_4NX2gmA-QVEVZPuotFCGvQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--NzB8fVQJ5HfG6fxh
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On 2019-Nov-10 22:16:26 -0330, Jonathan Anderson <jonathan.anderson@mun.ca>=
wrote:
>I=E2=80=99ve gone and done it: I upgraded a key ZFS-on-root machine from 1=
1.2 to
>12.0 and now I can't boot my ZFS-on-root pool. I wonder if the folks on
>this list might be able to help me figure out what's wrong and what I can
>do about it?
Based on your symptoms, it sounds like you might have a corrupt zpool.cache.
/boot/zfs/zpool.cache should be rewritten on every boot but I had one system
where that wasn't occurring and a FreeBSD upgrade (I don't currently recall
the actual versions) resulted in more thorough validation checks, which
failed.
Can you share your actual layout ("gpart show", "zpool status", details of
the non-partitioned disks, etc) - that might help us identify a problem.
>It looks like the ZFS code in the bootloader can't find anything in my root
>directory (zroot/ROOT/default), even though a booted FreeBSD kernel can. If
>I boot a rescue image from USB I can mount everything in the pool (`zpool
>import -f -R /mnt zroot`) and see all of my data, but when I run `lszfs
>zroot/ROOT/default` from the loader prompt it gives me an empty result (so,
>e.g., no /boot). Booting fails with messages such as, "i/o error - all blo=
ck
>copies unavailable".
If you boot from a rescue image and either delete (or rename) your
existing zpool.cache or run
# zpool set cachefile=3D/mnt/boot/zfs/zpool.cache zroot
(where the path cachefile path maps to /boot/zfs/zpool.cache at boot), does
that help?
>My pool consists of three mirrored vdevs, in which the first mirror uses G=
PT
> partitioning (for the boot partitions) and the other two mirrors use
>whole disks.
Whole disks are not recommended for anything other than building partitions
in. Ideally, you want all the disks to have GPT (or similar) partitions.
If you don't need anything else on the disk, just create a single partition
occupying the entire disk[1]. (I'd also recommend having a small root zpool
that is a single, preferably mirrored, vdev, rather than a large root spread
over multiple vdevs).
>I recall reading somewhere that the bootloader ZFS code doesn't like
>non-partition-based
>vdevs... is that true? If so, perhaps the issue is that my upgrade caused
>/boot to live on one of the newer whole-disk-based mirrors, hiding it from
>the bootloader's view?
That might be possible though it's not clear why it wouldn't have caused a
problem in the past. Note that the bootloader is performing I/O via the
BIOS so if the BIOS can't see the disks, the bootloader won't be able to
read them.
>> partitions with bootcode and file systems needed for booting can be
>added. This allows booting from disks that are also members of a pool.
>There is no performance penalty on FreeBSD when using a partition rather
>than a whole disk
Actually, Solaris might insist that you add a whole disk but, under
the hood, ZFS actually partitions the disk with a single partition
occupying the entire disk.
>The Handbook suggests that it's possible to break disks into multiple
>partitions that are added separately to a vdev, but is this a sensible
>thing to do? Is there any other ZFS lore that hasn't made it to the
>Handbook but that ought to be kept in mind from the outset?
You can have multiple partitions on a disk and put different partitions
into different vdevs but there's no point in having different partitions
on the same disk in the same vdev - that will reduce both performance
and resilience.
[1] Though leaving a free GB or so can help if you need to replace a disk a=
nd
the replacement is slightly smaller.
--=20
Peter Jeremy
--NzB8fVQJ5HfG6fxh
Content-Type: application/pgp-signature; name="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAl3I+0BfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF
QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi
CzQ6yhAAj/hYwvytdS2IdtZfQtBcX29j2h5FRfIBMwBBkZJnB6hekCiOgGhPAuL0
D+FNAi1GY50NSe2qcgxvVOGqKlGcz5f8xoRUWlYg4u1y+cb1XwEXU87ujW4BoDWd
kxnZaYozeB9IANa5FZwl9BdFcYTNIUzWA9Z2hc3ofCCEgG9Ckl+cnjlQNRXdbQAv
cZEMof/mqNYfRtEXPQboAOcB5MVdZu3ytn/s5tlI2Nk9EiWyJ+zue6UZfBuTfraI
13Zt6AVWgXedyD1zDToAl5bfexlVuE8gjVZH3FOcGNfQgYvMUnnkzrK2/25vH3cJ
0rvSN53DU5zTIdyoI3xP80ck416gF/sVK/v5CyPmHdNkS1cqIHsCLfEOCfh+9PtP
fNULo3LQf/cuY4SG8CkCh097NJ3Fq6X+3dHqlRFJbIkoShhCZNvx1XNXmy2KOI6M
wlGi52M+PdDcU8hrUGYURlwbhn1dtMxSQTMIqLHq5tq15ivPaCi4lOLlDwBwZBE9
BfIV331gm8KcfRkW3H4qcbS8gIEsubuKTsIyJibLkxfNT0XUPNT8SrRaT9rKsdyN
rsxTTg177E/PjE9awLSnOWVzhUjFy6e4Ml+YIZnSKWAudi6xVjyPO5MjUIdP+IYP
+eJ+4ddRGjPIC0nfyQqx57InXyoQvDdOv8VdfSgC+mxVcWJB98k=
=mbBZ
-----END PGP SIGNATURE-----
--NzB8fVQJ5HfG6fxh--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20191111061015.GA50716>