From owner-freebsd-stable@FreeBSD.ORG Mon Feb 13 16:11:35 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95C2C106564A for ; Mon, 13 Feb 2012 16:11:35 +0000 (UTC) (envelope-from O.Seibert@cs.ru.nl) Received: from rustug.science.ru.nl (rustug.science.ru.nl [131.174.16.158]) by mx1.freebsd.org (Postfix) with ESMTP id 138868FC13 for ; Mon, 13 Feb 2012 16:11:34 +0000 (UTC) Received: from kookpunt.science.ru.nl (kookpunt.science.ru.nl [131.174.30.61]) by rustug.science.ru.nl (8.13.7/5.31) with ESMTP id q1DFx6Db015470 for ; Mon, 13 Feb 2012 16:59:06 +0100 (MET) Received: from twoquid.cs.ru.nl (twoquid.cs.ru.nl [131.174.142.38]) by kookpunt.science.ru.nl (8.13.7/5.31) with ESMTP id q1DFx3MI014283; Mon, 13 Feb 2012 16:59:03 +0100 (MET) Received: by twoquid.cs.ru.nl (Postfix, from userid 4100) id F0AE62E04A; Mon, 13 Feb 2012 16:59:02 +0100 (CET) Date: Mon, 13 Feb 2012 16:59:02 +0100 From: Olaf Seibert To: freebsd-stable@freebsd.org Message-ID: <20120213155902.GE867@twoquid.cs.ru.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.19 (2009-01-05) X-Spam-Score: -1.799 () ALL_TRUSTED,BAYES_50 X-Scanned-By: MIMEDefang 2.63 on 131.174.30.61 Subject: ZFS faulted pool problem X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Feb 2012 16:11:35 -0000 Hi there, I run in production a FreeBSD-8 stable server with a ZFS file system. It used raidz2, so that ought to be very safe, I thought. Recently a disk failed, and it was replaced today. However, the file pool doesn't want to come back. Below is the status. da4 was bad and was replaced (by a somewhat bigger disk) in-place. The raidz2 shows to be DEGRADED but somehow the pool as a whole is FAULTED? And whatever I do, the new disk isn't recognised. I cut and pasted some commands below. In all cases it complains that pool "tank" is unavailable... What to do? No I/O errors show up in /var/log/messages. Previously, I had some weird directory where there were CRC errors in the directory entries, and as long as you didn't try to do anything with the files, there was no problem. Removing them was impossible though. I hope it isn't that what is holding my whole pool hostage... I don't mind losing that one directory. fourquid.1:/lib$ sudo zpool status Password: pool: tank state: FAULTED status: One or more devices could not be opened. There are insufficient replicas for the pool to continue functioning. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-3C scan: scrub repaired 0 in 49h3m with 2 errors on Fri Jan 20 15:10:35 2012 config: NAME STATE READ WRITE CKSUM tank FAULTED 0 0 2 raidz2-0 DEGRADED 0 0 8 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 3758301462980058947 UNAVAIL 0 0 0 was /dev/da4 da5 ONLINE 0 0 0 fourquid.1:/lib$ sudo zpool replace tank da4 cannot open 'tank': pool is unavailable fourquid.1:/lib$ sudo zpool replace raidz2-0 da4 cannot open 'raidz2-0': no such pool fourquid.1:/lib$ sudo zpool replace tank da4 da4 cannot open 'tank': pool is unavailable fourquid.1:/lib$ sudo zpool online tank da4 cannot open 'tank': pool is unavailable fourquid.1:/lib$ sudo zpool clear tank da4 cannot clear errors for da4: I/O error fourquid.1:/lib$ sudo zpool clear tank cannot clear errors for tank: I/O error fourquid.1:/lib$ sudo zpool replace tank 3758301462980058947 da4 cannot open 'tank': pool is unavailable fourquid.1:/lib$ sudo zpool scrub tank cannot scrub 'tank': pool is currently unavailable -Olaf. -- Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe));