From owner-freebsd-fs@FreeBSD.ORG Mon Jan 9 19:26:55 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9BAE81065680 for ; Mon, 9 Jan 2012 19:26:55 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from smtp.timeweb.ru (smtp.timeweb.ru [92.53.116.57]) by mx1.freebsd.org (Postfix) with ESMTP id 54A618FC1C for ; Mon, 9 Jan 2012 19:26:55 +0000 (UTC) Received: from [213.148.20.85] (helo=hive.panopticon) by smtp.timeweb.ru with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76) (envelope-from ) id 1RkKRe-0003mH-BY for freebsd-fs@FreeBSD.org; Mon, 09 Jan 2012 22:59:46 +0400 Received: from hades.panopticon (hades.panopticon [192.168.0.32]) by hive.panopticon (Postfix) with ESMTP id 6B223B84D for ; Mon, 9 Jan 2012 22:59:45 +0400 (MSK) Received: by hades.panopticon (Postfix, from userid 1000) id 14DF891; Mon, 9 Jan 2012 22:59:45 +0400 (MSK) Date: Mon, 9 Jan 2012 22:59:45 +0400 From: Dmitry Marakasov To: freebsd-fs@FreeBSD.org Message-ID: <20120109185944.GA8140@hades.panopticon> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Subject: Issues with multiple-vdev ZFS root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2012 19:26:55 -0000 Hi! I've recently moved to ZFS root installation with 2-way mirror with an intent to add another pair of disks later, however when I've actually tried to add second vdev, I've got "root pool can not have multiple vdevs or separate logs" error. It's possible to fool this error by removing bootfs property from the pool, adding second vdev and the re-adding the property. I've decided to first experiment in virtualbox. It turned out that after adding second mirror this way, system was still bootable. However, after copying /boot over (mv /boot /boot.old && cp -RPp /boot.old /boot) the system won't boot with ZFS: i/o error - all block copies unavailable Invalid format error (see [1]). It seems like the files have landed onto second vdev and gptzfsboot is no longer able to get them. Still, I've managed to solve that by increasing number of copies for the pool to 3 and copying /boot over again, which likely made the files available from first vdev again. Later I was hinted that the cause of this is that virtualbox BIOS only reports first disk, so the whole second vdev is unavailable for gptzfsboot. My question is whether these conclusions are correct: - FreeBSD supports any configuration for root zfs pool, including one with multiple vdevs. - However, one should ensure that BIOS reports all disks or increase number of copies for bootable dataset (and this should probably be documented somewhere) - The 'root pool can not have multiple vdevs' check should be removed [1] http://amdmi3.ru/files/zfs-boot-fail.png -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amdmi3@amdmi3.ru ..: jabber: amdmi3@jabber.ru http://www.amdmi3.ru