From owner-freebsd-bugs@FreeBSD.ORG Wed Apr 18 16:10:05 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCFF9106564A for ; Wed, 18 Apr 2012 16:10:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id B1A7E8FC15 for ; Wed, 18 Apr 2012 16:10:05 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3IGA5dg092322 for ; Wed, 18 Apr 2012 16:10:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3IGA53G092321; Wed, 18 Apr 2012 16:10:05 GMT (envelope-from gnats) Resent-Date: Wed, 18 Apr 2012 16:10:05 GMT Resent-Message-Id: <201204181610.q3IGA53G092321@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Peter Maloney Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 527E4106564A for ; Wed, 18 Apr 2012 16:01:27 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 3D8888FC08 for ; Wed, 18 Apr 2012 16:01:27 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3IG1QSJ078907 for ; Wed, 18 Apr 2012 16:01:26 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q3IG1PSZ078904; Wed, 18 Apr 2012 16:01:25 GMT (envelope-from nobody) Message-Id: <201204181601.q3IG1PSZ078904@red.freebsd.org> Date: Wed, 18 Apr 2012 16:01:25 GMT From: Peter Maloney To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: misc/167065: [zfs] boot fails when a spare is the boot disk X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Apr 2012 16:10:06 -0000 >Number: 167065 >Category: misc >Synopsis: [zfs] boot fails when a spare is the boot disk >Confidential: no >Severity: non-critical >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Wed Apr 18 16:10:05 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Peter Maloney >Release: 8-STABLE Feb 4 2012 >Organization: Brockmann Consult >Environment: FreeBSD bcnas1bak.bc.local 8.2-STABLE-20120204 FreeBSD 8.2-STABLE-20120204 #1: Fri Feb 10 15:35:01 CET 2012 root@bcnas1bak.bc.local:/usr/obj/usr/src/sys/GENERIC amd64 >Description: The story: A disk faulted. It was the first disk in the chassis (so it is the one the controller selects to boot). I added a spare (as the 2nd disk in the chassis), and replaced the faulted disk with the spare. Then I got the system to hang for unrelated reasons, removed the bad disk, and rebooted. (unrelated reasons: I added some zpool v16 zfs v4 disks I created on FreeBSD, tried mouting in Linux with zfs-fuse, which told me a device was missing, and then put them back in the FreeBSD and imported, which also said a device was missing, and then "zpool import -m -F -N" or something similar made it hang. parted on Linux said the disks were ext3, which may be related to the problem.) And on boot, to my surprise, it complained saying I can only boot from mirrors, raidz, etc. (a list which did not include spares). The second disk was defective in a different way, so I didn't try putting it first in the chassis, but that is a workaround, not a bug fix. >How-To-Repeat: Create a virtual machine with disks in this order: boot mirror disk 1 spare disk boot mirror disk 2 Boot it up. (using gpt slices, matching what I set up before). Replace the boot mirror disk 1 with the spare with: # gpart create -s gpt da1 # gpart add -b 34 -s 128 -t freebsd-boot da1 # gpart add -b 129024 -s 32768 -t freebsd-swap da1 # gpart add -b 258048 -s 8386560 -t freebsd-zfs -l sp_root0 da1 # zpool add zroot spare gpt/sp_root0 # zpool replace zroot gpt/root0 gpt/sp_root0 # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1 Wait for resilver. $ zpool status zroot ------------------------------------------------- pool: zroot state: ONLINE scan: resilvered 1.33G in 0h5m with 0 errors on Wed Apr 18 19:42:19 2012 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 spare-0 ONLINE 0 0 0 gpt/root0 ONLINE 0 0 0 gpt/sp_root0 ONLINE 0 0 0 gpt/root1 ONLINE 0 0 0 spares 665168513187118073 INUSE was /dev/gpt/sp_root0 errors: No known data errors ------------------------------------------------- # shutdown -p now Remove disk 1 and 3 (the old boot disks). Start up again. During boot, you get this output: ------------------------------------------------- ZFS: can only boot from disk, mirror, raidz1, raidz2 and raidz3 vdevs ZFS: i/o error - all block copies unavailable ZFS: can't read MOS ZFS: unexpected object set type 0 ZFS: unexpected object set type 0 FreeBSD/x86 boot Default: zroot:/boot/kernel/kernel boot: ZFS: unexpected object set type 0 FreeBSD/x86 boot Default: zroot:/boot/kernel/kernel boot: ------------------------------------------------- >Fix: Workaround was to boot the DVD, load the zfs modules, and then "zpool detach" the bad disk (converting the spare to a normal mirror), and detach the other root disk, which I wanted to remove anyway (to upgrade firmware). >Release-Note: >Audit-Trail: >Unformatted: