From owner-freebsd-fs@FreeBSD.ORG Thu Dec 24 00:36:06 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F2FB61065672 for ; Thu, 24 Dec 2009 00:36:06 +0000 (UTC) (envelope-from stevenschlansker@gmail.com) Received: from mail-yx0-f171.google.com (mail-yx0-f171.google.com [209.85.210.171]) by mx1.freebsd.org (Postfix) with ESMTP id A88168FC16 for ; Thu, 24 Dec 2009 00:36:06 +0000 (UTC) Received: by yxe1 with SMTP id 1so7351686yxe.3 for ; Wed, 23 Dec 2009 16:36:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:content-type:mime-version :subject:from:in-reply-to:date:content-transfer-encoding:message-id :references:to:x-mailer; bh=CCdAR7SPSjaMql8OnjuX6t2rt1UDEHPty9k3kynTZR8=; b=MA8ZjdwF+8A8FKLJ7y2LOM735uVKI6nvXzdNj80R260LEMWqZrI2XUWteHPSHrL3ea lKp8lt3/YFefTW+VF0PEqjqY/hqBtgw0gWJAnCFEdnicLx1pKdqLzli1EfHdaouvni9D mM22mzOOJvaT2rmDUUX1sNJRTSspG8ppXPtZg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; b=W5TOsqin3MHa8i64H3UKFr98pTq8XKzyO6cgYJVWpFyA0UaKPP8KzdE6id2W4I+GLI P5ev7kEMcQUbeJNc3TeOmADVCwzSXo3owiUG5+FC6X2Hn09Kmtv+ljwCSCn1Tp9s5NR5 2TNvWKKNd62ZJ58Ut4wYlHScCZhbnA+4QORTk= Received: by 10.100.24.9 with SMTP id 9mr6743445anx.186.1261614966176; Wed, 23 Dec 2009 16:36:06 -0800 (PST) Received: from 68-29-245-15.pools.spcsdns.net (68-29-245-15.pools.spcsdns.net [68.29.245.15]) by mx.google.com with ESMTPS id 20sm3287122yxe.20.2009.12.23.16.36.04 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 23 Dec 2009 16:36:05 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1077) From: Steven Schlansker In-Reply-To: <5da0588e0912221741r48395defnd11e34728d2b7b97@mail.gmail.com> Date: Wed, 23 Dec 2009 16:36:00 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <9CEE3EE5-2CF7-440E-B5F4-D2BD796EA55C@gmail.com> References: <048AF210-8B9A-40EF-B970-E8794EC66B2F@gmail.com> <4B315320.5050504@quip.cz> <5da0588e0912221741r48395defnd11e34728d2b7b97@mail.gmail.com> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.1077) Subject: Re: ZFS: Can't repair raidz2 (Cannot replace a replacing device) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2009 00:36:07 -0000 On Dec 22, 2009, at 5:41 PM, Rich wrote: > http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763 may be > useful to you - it's what we did when we got stuck in a resilver loop. > I recall being in the same state you're in right now at one point, and > getting out of it from there. >=20 > I think if you apply that patch, you'll be able to cancel the > resilver, and then resilver again with the device you'd like to > resilver with. >=20 Thanks for the suggestion, but the problem isn't that it's stuck in a resilver loop (which is what the patch seems to try to avoid) but that I can't detach a drive. Now I got clever and fudged a label onto the new drive (copied the first 50MB of one of the dying drives), ran a scrub, and have this layout - pool: universe state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are = unaffected. action: Determine if the device needs to be replaced, and clear the = errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scrub: scrub completed after 20h58m with 0 errors on Wed Dec 23 = 11:36:43 2009 config:=20 NAME STATE READ WRITE CKSUM universe DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 ad16 ONLINE 0 0 0 replacing DEGRADED 0 0 40.7M ad26 ONLINE 0 0 0 506G = repaired 6170688083648327969 UNAVAIL 0 88.7M 0 was = /dev/ad12 ad8 ONLINE 0 0 0 concat/back2 ONLINE 0 0 0 ad10 ONLINE 0 0 0 concat/ad4ex ONLINE 0 0 0 ad24 ONLINE 0 0 0 concat/ad6ex ONLINE 48 0 0 28.5K = repaired Why has the replacing vdev not gone away? I still can't detach - [steven@universe:~]% sudo zpool detach universe 6170688083648327969 cannot detach 6170688083648327969: no valid replicas even though now there actually is a valid replica (ad26) Additionally, running zpool clear hangs permanently and in fact freezes = all IO to the pool. Since I've mounted /usr from the pool, this is effectively death to the system. Any other zfs commands seem to work okay (zpool scrub, zfs mount, etc.). Just clear is insta-death. I can't help but suspect that this is caused by the now non-sensical vdev = configuration (replacing with one good drive and one nonexistent one)... Any further thoughts? Thanks, Steven > - Rich >=20 > On Tue, Dec 22, 2009 at 6:15 PM, Miroslav Lachman <000.fbsd@quip.cz> = wrote: >> Steven Schlansker wrote: >>>=20 >>> As a corollary, you may notice some funky concat business going on. >>> This is because I have drives which are very slightly different in = size (< >>> 1MB) >>> and whenever one of them goes down and I bring the pool up, it = helpfully >>> (?) >>> expands the pool by a whole megabyte then won't let the drive back = in. >>> This is extremely frustrating... is there any way to fix that? I'm >>> eventually going to keep expanding each of my drives one megabyte at = a >>> time >>> using gconcat and space on another drive! Very frustrating... >>=20 >> You can avoid it by partitioning the drives to the well known = 'minimal' size >> (size of smallest disk) and use the partition instead of raw disk. >> For example ad12s1 instead of ad12 (if you creat slices by fdisk) >> of ad12p1 (if you creat partitions by gpart) >>=20 >> You can also use labels instead of device name. >>=20 >> Miroslav Lachman >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>=20 >=20 >=20 >=20 > --=20 >=20 > If you are over 80 years old and accompanied by your parents, we will > cash your check.