From owner-freebsd-fs@FreeBSD.ORG Sun Apr 15 11:20:24 2007 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8747416A400 for ; Sun, 15 Apr 2007 11:20:24 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id D637213C44C for ; Sun, 15 Apr 2007 11:20:23 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id AA091456AB; Sun, 15 Apr 2007 13:20:21 +0200 (CEST) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 4780E45685; Sun, 15 Apr 2007 13:20:16 +0200 (CEST) Date: Sun, 15 Apr 2007 13:19:55 +0200 From: Pawel Jakub Dawidek To: Barry Pederson Message-ID: <20070415111955.GB16971@garage.freebsd.pl> References: <46205338.3090803@barryp.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="eJnRUKwClWJh1Khz" Content-Disposition: inline In-Reply-To: <46205338.3090803@barryp.org> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@FreeBSD.org Subject: Re: ZFS raidz device replacement problem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Apr 2007 11:20:24 -0000 --eJnRUKwClWJh1Khz Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Apr 13, 2007 at 11:06:16PM -0500, Barry Pederson wrote: > I've been playing with ZFS (awesome stuff, thanks PJD) and noticed someth= ing funny when replacing a device under a raidz pool. It seems that even t= hough ZFS says=20 > resilvering is complete, you still need to manually do a "zpool scrub" to= really get the pool into a good state. How do you tell it's not in a good state? > From what I've read in the "Solaris ZFS Administration Guide", it doesn't= seem that that step should be required. Is there some kind of auto-scrub = being missed? >=20 > I've tried to show this below with some md devices, creating 4 of them, p= utting 3 into a raidz and then replacing one. This is with a world and ker= nel csupped and built=20 > earlier today (2007-04-13) [...] > # zpool create mypool raidz md0 md1 md2 [...] > # zpool replace mypool md2 md3 [...] > # zpool status mypool > pool: mypool > state: ONLINE > scrub: resilver completed with 0 errors on Fri Apr 13 22:43:19 2007 > config: >=20 > NAME STATE READ WRITE CKSUM > mypool ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > md0 ONLINE 0 0 0 > md1 ONLINE 0 0 0 > replacing ONLINE 0 0 0 > md2 ONLINE 0 0 0 > md3 ONLINE 0 0 0 Do you mean that resilver is completed, but it is still replacing? ZFS resilvers only live data, those md's was free, so it completed immediately. > # zpool status mypool > pool: mypool > state: ONLINE > scrub: resilver completed with 0 errors on Fri Apr 13 22:43:19 2007 > config: >=20 > NAME STATE READ WRITE CKSUM > mypool ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > md0 ONLINE 0 0 0 > md1 ONLINE 0 0 0 > md3 ONLINE 0 0 0 It looks everything is fine. What's wrong? > # zpool scrub mypool >=20 > # zpool status mypool > pool: mypool > state: ONLINE > status: One or more devices has experienced an unrecoverable error. An > attempt was made to correct the error. Applications are unaffect= ed. > action: Determine if the device needs to be replaced, and clear the errors > using 'zpool clear' or replace the device with 'zpool replace'. > see: http://www.sun.com/msg/ZFS-8000-9P > scrub: scrub completed with 0 errors on Fri Apr 13 22:43:46 2007 > config: >=20 > NAME STATE READ WRITE CKSUM > mypool ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > md0 ONLINE 0 0 0 > md1 ONLINE 0 0 0 > md3 ONLINE 0 0 5 If you are referring to this CKSUM count not beeing 0, this was a bug in ZFS itself, and was fixes in OpenSolaris already and fix was merged to FreeBSD. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --eJnRUKwClWJh1Khz Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD4DBQFGIgpbForvXbEpPzQRAjhdAJd4VOMEq5V6tzfd7vIOK2gRmfh6AJ93BK2g fd0ANARFcae6ykqw53VYFg== =U0I+ -----END PGP SIGNATURE----- --eJnRUKwClWJh1Khz--