Date: Wed, 19 Nov 2014 18:17:50 +0100 (CET) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= <Trond.Endrestol@fagskolen.gjovik.no> To: Anton Eliasson <devel@antoneliasson.se> Cc: freebsd-questions@freebsd.org Subject: Re: Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file system after upgrade to 10.1-RELEASE Message-ID: <alpine.BSF.2.11.1411191756210.7150@mail.fig.ol.no> In-Reply-To: <546CC0F1.2050607@antoneliasson.se> References: <546BC2A5.6010806@antoneliasson.se> <alpine.BSF.2.11.1411190814400.7150@mail.fig.ol.no> <546CC0F1.2050607@antoneliasson.se>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 19 Nov 2014 17:10+0100, Anton Eliasson wrote: > Den 2014-11-19 08:22, Trond Endrestøl skrev: > > On Tue, 18 Nov 2014 23:05+0100, Anton Eliasson wrote: > > > > > Hi! > > > After upgrading a server from FreeBSD 10.0-RELEASE to 10.1-RELEASE it no > > > longer starts up. The bootloader shows its usual menu and then proceeds to > > > load zfs.ko, opensolaris.ko and some third module. Then kernel boots and > > > continues all the way to mountroot where it fails with the message > > > "Mounting > > > from zfs:zroot/ROOT/default failed with error 2: unknown file system." The > > > entire message: > > > https://cloud.solid.lth.se/public.php?service=files&t=27c15a131ca63b19f619a7ee558fe8d1 > > > > > > Entering zfs:zroot, zfs:zroot/ROOT, zfs:zroot/ROOT/default or > > > zfs:anythingelse > > > results in the same error. > > > > > > FreeBSD 10.0 was installed this summer. I believe I used the installer's > > > guided procedure for installing on a mirrored ZFS root. Each drive has a > > > GPT > > > table with a 512k freebsd-boot, a freebsd-swap and a freebsd-zfs > > > partition. No > > > separate /boot. > > > > > > There is no vfs.root.mountfrom directive in /boot/loader.conf and no / in > > > /etc/fstab. If I set vfs.root.mountfrom="zfs:zroot" it fails with the same > > > message but a slightly different path (zfs:zroot). > > > > > > The motherboard is a Supermicro X9SRH-7TF. I can import and mount the root > > > filesystem with a FreeBSD live CD without problems. > > > > > > Some config files pulled from a very similar installation (same server > > > model, > > > installed at the same time). They should be more or less identical on the > > > server that won't boot: > > > > > > /etc/rc.conf: > > > > > > hostname="..." > > > keymap="swedish.iso.kbd" > > > # Use synchronous DHCP; pause boot until DHCP is completed. > > > ifconfig_ix0="SYNCDHCP" > > > sshd_enable="YES" > > > ntpd_enable="YES" > > > # Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable > > > dumpdev="AUTO" > > > zfs_enable="YES" > > > > > > # IPFW > > > [...] > > > > > > /boot/loader.conf: > > > > > > zfs_load="YES" > > > geom_mirror_load="YES" > > > > > > /etc/fstab: > > > > > > fdesc /dev/fd fdescfs rw 0 0 > > > /dev/mirror/swap none swap sw 0 0 > > > > > > Thanks! > > According to the screenshot, the bootfs property on zroot is set to > > zroot/ROOT/default. Does that dataset exist? > > > > While at the loader prompt, what does these commands tell you? > > > > lszfs zroot > $MOS > $FREE > $ORIGIN > ROOT > tmp > usr > var > > lszfs zroot/ROOT > default > > And `lszfs zroot/ROOT/default` shows nothing. > > If you see some other potential dataset, try setting currdev using > > this as a template before booting: > > > > set currdev="zfs:zroot/ROOT/some-other-dataset:" > > ^ ^ > > These :'s are important. > `show currdev` shows that it is already set to zfs:zroot/ROOT/default: (with a > colon at the end) > > > > Next, unload the old stuff, and try to load the new stuff: > > > > unload > > load /boot/kernel/kernel > > load /boot/kernel/opensolaris.ko > > load /boot/kernel/zfs.ko > It looks like it loads the kernel, but then it fails on the following two > commands because the kernel is not loaded: > https://cloud.solid.lth.se/public.php?service=files&t=5028527feac7982c76206b691212d15b > https://cloud.solid.lth.se/public.php?service=files&t=0c0d50aac583167a11a8eaa4de5f08ce > > Doing boot -s after that gives the same error as before. > > However, loading the old kernel with the following commands works better: > > unload > load /boot/kernel.old/kernel > load /boot/kernel.old/opensolaris.ko > load /boot/kernel/zfs.ko > lsmod > > Screenshots: > https://cloud.solid.lth.se/public.php?service=files&t=9c6c1bbc96b2c816aae05d2718215e2a > https://cloud.solid.lth.se/public.php?service=files&t=8aee8df4bacf65b9ceae02fa65b53711 > > Issuing boot -s after that shows some text and then suddenly reboots the > computer. I can see "ZFS filesystem version 5" in the output so it looks like > it is successfully loaded. To break into the kernel debugger, enter this before booting the kernel: set debug.debugger_on_panic="1" Judging from the screenshots you've provided and from what you wrote in your reply, it seems vfs.root.mountfrom is overridden somehow. /boot/loader.conf would be a natural candidate. If the bootfs property is indeed set to zroot/ROOT/default, then it's very strange that the kernel would proclaim: Trying to mount root from zfs:zroot []... instead of: Trying to mount root from zfs:zroot/ROOT/default []... And it's even stranger that nothing appears on lszfs' radar when asked to list zroot/ROOT/default. > Selecting the old kernel in the boot menu and trying to boot normally fails > because init is not found: > https://cloud.solid.lth.se/public.php?service=files&t=81fd35bb6348ae7426553c2f59fb2a18 > > Could this be a mismatch in kernel and kernel module versions? How would I fix > that? If you can boot from a recent snapshot image or live image, and import the zpool read-only, you could walk around and see if zroot/ROOT/default/boot/loader.conf contains anything strange. This zpool command would allow you to go spelunking inside /zroot: zpool import -o readonly=on -f -R /zroot zroot You might need to mount readonly any dataset you wish to examine. If you see the need for changing any of the files, you must export the zpool and re-import it readwrite, i.e. leave out the -o readonly=on option. I admit I'm in unknown territory, even if I have managed multiple ZFS systems over the past years, so be careful. -- +-------------------------------+------------------------------------+ | Vennlig hilsen, | Best regards, | | Trond Endrestøl, | Trond Endrestøl, | | IT-ansvarlig, | System administrator, | | Fagskolen Innlandet, | Gjøvik Technical College, Norway, | | tlf. mob. 952 62 567, | Cellular...: +47 952 62 567, | | sentralbord 61 14 54 00. | Switchboard: +47 61 14 54 00. | +-------------------------------+------------------------------------+ From owner-freebsd-questions@FreeBSD.ORG Wed Nov 19 17:47:31 2014 Return-Path: <owner-freebsd-questions@FreeBSD.ORG> Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E299A163 for <freebsd-questions@freebsd.org>; Wed, 19 Nov 2014 17:47:30 +0000 (UTC) Received: from smtp05.citynetwork.se (mail.citynetwork.se [IPv6:2a00:16d8:0:4::200]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 98B26A14 for <freebsd-questions@freebsd.org>; Wed, 19 Nov 2014 17:47:30 +0000 (UTC) Received: from localhost (smtp05.citynetwork.se [127.0.0.1]) by smtp05.citynetwork.se (Postfix) with ESMTP id 51A70801859; Wed, 19 Nov 2014 18:47:27 +0100 (CET) X-Virus-Scanned: amavisd-new at citynetwork.se Received: from smtp05.citynetwork.se ([127.0.0.1]) by localhost (smtp05.citynetwork.se [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h-o5hFYqW0hm; Wed, 19 Nov 2014 18:47:26 +0100 (CET) Received: from [172.20.10.13] (m83-185-159-168.cust.tele2.se [83.185.159.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: devel@antoneliasson.se) by smtp05.citynetwork.se (Postfix) with ESMTPSA id F296B801886; Wed, 19 Nov 2014 18:47:25 +0100 (CET) Message-ID: <546CD7AA.9040107@antoneliasson.se> Date: Wed, 19 Nov 2014 18:47:22 +0100 From: Anton Eliasson <devel@antoneliasson.se> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: =?windows-1252?Q?Trond_Endrest=F8l?= <Trond.Endrestol@fagskolen.gjovik.no> Subject: Re: Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file system after upgrade to 10.1-RELEASE References: <546BC2A5.6010806@antoneliasson.se> <alpine.BSF.2.11.1411190814400.7150@mail.fig.ol.no> <546CC0F1.2050607@antoneliasson.se> <alpine.BSF.2.11.1411191756210.7150@mail.fig.ol.no> In-Reply-To: <alpine.BSF.2.11.1411191756210.7150@mail.fig.ol.no> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions <freebsd-questions.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/>; List-Post: <mailto:freebsd-questions@freebsd.org> List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=subscribe> X-List-Received-Date: Wed, 19 Nov 2014 17:47:31 -0000 Den 2014-11-19 18:17, Trond Endrestøl skrev: > To break into the kernel debugger, enter this before booting the > kernel: > > set debug.debugger_on_panic="1" > > Judging from the screenshots you've provided and from what you wrote > in your reply, it seems vfs.root.mountfrom is overridden somehow. > /boot/loader.conf would be a natural candidate. > > If the bootfs property is indeed set to zroot/ROOT/default, then it's > very strange that the kernel would proclaim: > > Trying to mount root from zfs:zroot []... > > instead of: > > Trying to mount root from zfs:zroot/ROOT/default []... > > And it's even stranger that nothing appears on lszfs' radar when asked > to list zroot/ROOT/default. > >> Selecting the old kernel in the boot menu and trying to boot normally fails >> because init is not found: >> https://cloud.solid.lth.se/public.php?service=files&t=81fd35bb6348ae7426553c2f59fb2a18 >> >> Could this be a mismatch in kernel and kernel module versions? How would I fix >> that? > If you can boot from a recent snapshot image or live image, and import > the zpool read-only, you could walk around and see if > zroot/ROOT/default/boot/loader.conf contains anything strange. > > This zpool command would allow you to go spelunking inside /zroot: > > zpool import -o readonly=on -f -R /zroot zroot > > You might need to mount readonly any dataset you wish to examine. > > If you see the need for changing any of the files, you must export the > zpool and re-import it readwrite, i.e. leave out the -o readonly=on > option. > > I admit I'm in unknown territory, even if I have managed multiple ZFS > systems over the past years, so be careful. > Hi! I had forgot to remove the vfs.root.mountfrom that I experimented with. This probably caused the failure to load init. I got it to work by chrooting into the system from a live cd and rolling back the upgrade with `freebsd-update rollback`. Then it booted fine. Now freebsd-version shows 10.0-RELEASE-p12 and uname -r shows 10.1-RELEASE. It looks a bit strange, but at least it works for now. I may have mixed up the commands used to upgrade the system. Is the following the correct procedure for doing a major upgrade (with a stock kernel)? freebsd-update -r 10.1-RELEASE upgrade freebsd-update install (reboot) freebsd-update install (rebuild ports) (done) Thanks for your help! -- Best Regards, Anton Eliasson
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.11.1411191756210.7150>