From owner-freebsd-fs@freebsd.org Sun Oct 27 19:02:54 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B952D1A2D37 for ; Sun, 27 Oct 2019 19:02:54 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from blaine.gmane.org (195-159-176-226.customer.powertech.no [195.159.176.226]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 471S0Y240Qz41pP for ; Sun, 27 Oct 2019 19:02:52 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from list by blaine.gmane.org with local (Exim 4.89) (envelope-from ) id 1iOnoQ-0012qZ-7d for freebsd-fs@freebsd.org; Sun, 27 Oct 2019 20:02:50 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Johannes Totz Subject: zpool attach/resilver trashes bsdlabel bootcode Date: Sun, 27 Oct 2019 20:02:44 +0000 Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.2.0 X-Mozilla-News-Host: news://news://news.gmame.org:119 Content-Language: en-GB X-Rspamd-Queue-Id: 471S0Y240Qz41pP X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=jo-t.de (policy=none); spf=none (mx1.freebsd.org: domain of freebsd-fs@m.gmane.org has no SPF policy when checking 195.159.176.226) smtp.mailfrom=freebsd-fs@m.gmane.org X-Spamd-Result: default: False [4.61 / 15.00]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[jo-t.de : No valid SPF, No valid DKIM,none]; GREYLIST(0.00)[pass,body]; FROM_HAS_DN(0.00)[]; HFILTER_HOSTNAME_1(0.50)[195-159-176-226.customer.powertech.no]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(0.02)[0.020,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(0.24)[ipnet: 195.159.0.0/16(0.28), asn: 2116(0.94), country: NO(-0.01)]; NEURAL_SPAM_LONG(0.55)[0.550,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[johannes@jo-t.de,freebsd-fs@m.gmane.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2116, ipnet:195.159.0.0/16, country:NO]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; FROM_NEQ_ENVFROM(0.00)[johannes@jo-t.de,freebsd-fs@m.gmane.org]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Oct 2019 19:02:54 -0000 Hi there, how does a zpool attach and subsequent resilver deal with the bootcode that's inside a bsdlabel partition? I'm trying to create a bootable image (with mbr, slices, and partition) but every time I attach the partition to a pool the bootcode ends up garbled and the image is not bootable. For example: > /usr/bin/truncate -s 1025m /var/tmp/usbbootpoolimg.raw > /sbin/mdconfig -a -t vnode -u 3 -S 512 -y 16 -x 63 -f /var/tmp/usbbootpoolimg.raw > > /sbin/gpart create -s mbr /dev/md3 > /sbin/gpart add -s 1g -t freebsd /dev/md3 > /sbin/gpart set -a active -i 1 /dev/md3 > /sbin/gpart bootcode -b /boot/boot0 /dev/md3 > /sbin/gpart create -s bsd /dev/md3s1 > /sbin/gpart add -t freebsd-zfs /dev/md3s1 > > /sbin/bsdlabel -B /dev/md3s1 > /bin/dd if=/boot/zfsboot of=/var/tmp/zfsboot1 count=1 > /sbin/gpart bootcode -b /var/tmp/zfsboot1 /dev/md3s1 > /bin/dd if=/boot/zfsboot of=/dev/md3s1a skip=1 seek=1024 All good up to here. Image is bootable, but there is no pool yet in the image so the bootloader complains. Fine. Then: > /sbin/zpool attach bootpool gpt/boot2 /dev/md3s1a Now the freshly installed bootcode is trashed. If I try to boot this it makes it to the F1 boot prompt (the mbr part) but then hangs indefinitely. If I write the bootcode again, after the pool resilver is done then the image boots. To me, the bootcode is part of the partitioning scheme, not part of the pool that needs resilvering. Any ideas? Cheers, Johannes From owner-freebsd-fs@freebsd.org Mon Oct 28 23:54:51 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C260D1604D4 for ; Mon, 28 Oct 2019 23:54:51 +0000 (UTC) (envelope-from johannes@jo-t.de) Received: from email.jo-t.de (seppel.jo-t.de [188.68.55.77]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 472BQy4cCCz4fKd; Mon, 28 Oct 2019 23:54:49 +0000 (UTC) (envelope-from johannes@jo-t.de) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=jo-t.de; s=jot0; t=1572306882; bh=hkRW1856O0NolgvXtqNmzVuHCs3M4Qg4XZOyu9TwbhI=; h=Subject:To:References:From:Message-ID:Date:MIME-Version:From; b=6hUdOrERClAJ4/7DDsB5MEDQBveEQBng54kUpkopKP9b9L7IJPthoQf7i5phTIl4O aJVR6gwgjOPZXXRZ2YhlcygzZZoebVcIwj0UnNof2+vuRFeUhEpbr/6h3WOR0MToD2 JCAvnUliVraAbyA6obsqmA6OJVON+V9/FxH9h1nI76GSgho9Iq0TdH4wIPQlANLQJ3 lq7Z9CzmkcyHOKHd5yTzXBy2/Z1jrwFZMmpX2KZgieKhBaxRatJyTKh2NAXLQyrYt6 16uCc+5MYAQDhlzOZDk+ZSZdToD+5htZnx98TMyXzub8ksYhnLkLodh/GBZdO0N2cS OxnHGv4pDjeOA== Subject: Re: zpool attach/resilver trashes bsdlabel bootcode To: Andriy Gapon , freebsd-fs@freebsd.org References: From: Johannes Totz Message-ID: <2ca27aa3-bec8-db97-20fb-bfdc37ad7ef7@jo-t.de> Date: Mon, 28 Oct 2019 23:54:41 +0000 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 472BQy4cCCz4fKd X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jo-t.de header.s=jot0 header.b=6hUdOrER; dmarc=pass (policy=none) header.from=jo-t.de; spf=pass (mx1.freebsd.org: domain of johannes@jo-t.de designates 188.68.55.77 as permitted sender) smtp.mailfrom=johannes@jo-t.de X-Spamd-Result: default: False [-3.12 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[jo-t.de:s=jot0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DKIM_TRACE(0.00)[jo-t.de:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[jo-t.de,none]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.12)[asn: 197540(-0.58), country: DE(-0.01)]; ASN(0.00)[asn:197540, ipnet:188.68.48.0/21, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Oct 2019 23:54:51 -0000 On 28/10/2019 13:06, Andriy Gapon wrote: > On 27/10/2019 22:02, Johannes Totz wrote: >> Hi there, >> >> how does a zpool attach and subsequent resilver deal with the bootcode that's >> inside a bsdlabel partition? >> I'm trying to create a bootable image (with mbr, slices, and partition) but >> every time I attach the partition to a pool the bootcode ends up garbled and the >> image is not bootable. >> >> For example: >>> /usr/bin/truncate -s 1025m /var/tmp/usbbootpoolimg.raw >>> /sbin/mdconfig -a -t vnode -u 3 -S 512 -y 16 -x 63 -f /var/tmp/usbbootpoolimg.raw >>> >>> /sbin/gpart create -s mbr /dev/md3 >>> /sbin/gpart add -s 1g -t freebsd /dev/md3 >>> /sbin/gpart set -a active -i 1 /dev/md3 >>> /sbin/gpart bootcode -b /boot/boot0 /dev/md3 >>> /sbin/gpart create -s bsd /dev/md3s1 >>> /sbin/gpart add -t freebsd-zfs /dev/md3s1 >>> >>> /sbin/bsdlabel -B /dev/md3s1 >>> /bin/dd if=/boot/zfsboot of=/var/tmp/zfsboot1 count=1 >>> /sbin/gpart bootcode -b /var/tmp/zfsboot1 /dev/md3s1 >>> /bin/dd if=/boot/zfsboot of=/dev/md3s1a skip=1 seek=1024 >> >> All good up to here. Image is bootable, but there is no pool yet in the image so >> the bootloader complains. Fine. >> Then: >> >>> /sbin/zpool attach bootpool gpt/boot2 /dev/md3s1a >> >> Now the freshly installed bootcode is trashed. If I try to boot this it makes it >> to the F1 boot prompt (the mbr part) but then hangs indefinitely. >> >> If I write the bootcode again, after the pool resilver is done then the image >> boots. >> >> To me, the bootcode is part of the partitioning scheme, not part of the pool >> that needs resilvering. >> Any ideas? > > You write the second part of zfsboot inside of ZFS managed area. > So, you should do that *after* ZFS initializes the partition. I had https://www.freebsd.org/cgi/man.cgi?query=zfsboot interpreted to mean that zfs wont touch the special boot area (even though it's under zfs control). Is that vdev initialisation the only time that the bootcode gets overwritten? Never on "normal" resilver? Cheers, Johannes