From owner-freebsd-fs@FreeBSD.ORG Sat Dec 6 01:54:59 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 99BB7996 for ; Sat, 6 Dec 2014 01:54:59 +0000 (UTC) Received: from mail-wi0-f173.google.com (mail-wi0-f173.google.com [209.85.212.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 211629F3 for ; Sat, 6 Dec 2014 01:54:58 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id r20so344939wiv.6 for ; Fri, 05 Dec 2014 17:54:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=qlO0QHkrpNahIOc40UhdNgDsWmwu+Kb1aue1NevUwRo=; b=jQEWq9kgyaw548HJMyNd98OfbpYRm9NplD36FnbHRaaBuqvbK6iF8xrWtINAUIX8Dt v26RLIkG0A0v0cAR/CXak/h9JCsuSPaww+mHAb98R90qF74KR4UlEED+Nl3c2LH6bh3m UooK9+W+SZSV2jz21okLaM1BclRVl/sBePKrr6CRhCzs2ZFmnq0mQ30FJwzUZd9AKsNi LpNH0HiBk0ndoUQlnAyKkXQcCBVvVb1BIYDTaTw4EmzuneJxMVqvZnLRvjpobmMIxFz2 nz28QUmPr3QSlZjnFfaa+/+SH1wLDBFbka7xfN9tn3gUg+/vIyyNPUNv+CkUxtNLlfEt 7X1Q== X-Gm-Message-State: ALoCoQkqvTNcgEMKuq+GNrPxxLpdlrH77uLHht0G3z9YI3GGcVvhd4gMvXowuBfB0NqzwqJPPVnV X-Received: by 10.180.20.6 with SMTP id j6mr7961201wie.59.1417830421848; Fri, 05 Dec 2014 17:47:01 -0800 (PST) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by mx.google.com with ESMTPSA id h2sm174300wix.5.2014.12.05.17.47.00 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Dec 2014 17:47:01 -0800 (PST) Message-ID: <54825F9B.6000009@multiplay.co.uk> Date: Sat, 06 Dec 2014 01:44:59 +0000 From: Steven Hartland User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: ZFS weird issue... References: <54825E70.20900@sorbs.net> In-Reply-To: <54825E70.20900@sorbs.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Dec 2014 01:54:59 -0000 Formatting for you zpool list looks odd, making it not clear if its your spare which is unavailable or one of your pool drives. If its the spare, try zpool remove. On 06/12/2014 01:40, Michelle Sullivan wrote: > Here's what happened: > > LSI9260-16i, 16x3T SATA drives. 15 LDs each configured as single disk > RAID0 last drive a hot spare. > > All LDs configured as a ZFS RaidZ2 pool ('sorbs') > > Bay/Drive 8 failed, hot spare kicked in. ZFS resilvered. (mfid15 became > spare-8) > > On reboot mfid9-15 were re-named automatically to mfid8-14 ... ZFS > didn't seems to care... > > Days later new drive to replace the dead drive arrived and was > inserted. System refused to re-add as there was data in the cache, so > rebooted and cleared the cache (as per many on web faq's) Reconfigured > it to match the others. Can't do a zpool replace mfid8 because that's > already in the pool... (was mfid9) can't use mfid15 because zpool > reports it's not part of the config... can't use the uniq-id it received > (can't find vdev) ... HELP!! :) > > This is what I can see currently ... > > Thanks, in advance. > > root@colossus:~ # zpool status -v > pool: VirtualDisks > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > VirtualDisks ONLINE 0 0 0 > zvol/sorbs/VirtualDisks ONLINE 0 0 0 > > errors: No known data errors > > pool: sorbs > state: DEGRADED > status: One or more devices could not be opened. Sufficient replicas > exist for > the pool to continue functioning in a degraded state. > action: Attach the missing device and online it using 'zpool online'. > see: http://illumos.org/msg/ZFS-8000-2Q > scan: scrub in progress since Fri Dec 5 17:11:29 2014 > 2.51T scanned out of 29.9T at 89.4M/s, 89h7m to go > 0 repaired, 8.40% done > config: > > NAME STATE READ WRITE CKSUM > sorbs DEGRADED 0 0 0 > raidz2-0 DEGRADED 0 0 0 > mfid0 ONLINE 0 0 0 > mfid1 ONLINE 0 0 0 > mfid2 ONLINE 0 0 0 > mfid3 ONLINE 0 0 0 > mfid4 ONLINE 0 0 0 > mfid5 ONLINE 0 0 0 > mfid6 ONLINE 0 0 0 > mfid7 ONLINE 0 0 0 > spare-8 DEGRADED 0 0 0 > 1702922605 UNAVAIL 0 0 0 was /dev/mfid8 > mfid14 ONLINE 0 0 0 > mfid8 ONLINE 0 0 0 > mfid9 ONLINE 0 0 0 > mfid10 ONLINE 0 0 0 > mfid11 ONLINE 0 0 0 > mfid12 ONLINE 0 0 0 > mfid13 ONLINE 0 0 0 > spares > 933862663 INUSE was /dev/mfid14 > > errors: No known data errors > root@colossus:~ # uname -a > FreeBSD colossus.sorbs.net 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: > Thu Sep 26 22:50:31 UTC 2013 > root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > root@colossus:~ # sh ./lsi.sh drives > Slot Number: 0 - Online, Spun Up > Slot Number: 1 - Online, Spun Up > Slot Number: 2 - Online, Spun Up > Slot Number: 3 - Online, Spun Up > Slot Number: 4 - Online, Spun Up > Slot Number: 5 - Online, Spun Up > Slot Number: 6 - Online, Spun Up > Slot Number: 7 - Online, Spun Up > Slot Number: 8 - Online, Spun Up > Slot Number: 9 - Online, Spun Up > Slot Number: 10 - Online, Spun Up > Slot Number: 11 - Online, Spun Up > Slot Number: 12 - Online, Spun Up > Slot Number: 13 - Online, Spun Up > Slot Number: 14 - Online, Spun Up > Slot Number: 15 - Online, Spun Up > root@colossus:~ # sh ./lsi.sh status > > > Adapter 0 -- Virtual Drive Information: > Virtual Drive: 0 (Target Id: 0) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 1 (Target Id: 1) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 2 (Target Id: 2) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 3 (Target Id: 3) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 4 (Target Id: 4) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 5 (Target Id: 5) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 6 (Target Id: 6) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 7 (Target Id: 7) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 8 (Target Id: 8) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 512 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Disk's Default > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 9 (Target Id: 9) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 10 (Target Id: 10) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 11 (Target Id: 11) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 12 (Target Id: 12) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 13 (Target Id: 13) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 14 (Target Id: 14) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 64 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > Virtual Drive: 15 (Target Id: 15) > Name : > RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 > Size : 2.728 TB > Sector Size : 512 > Is VD emulated : Yes > Parity Size : 0 > State : Optimal > Strip Size : 256 KB > Number Of Drives : 1 > Span Depth : 1 > Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if > Bad BBU > Default Access Policy: Read/Write > Current Access Policy: Read/Write > Disk Cache Policy : Enabled > Encryption Type : None > Bad Blocks Exist: No > Is VD Cached: No > > > > Exit Code: 0x00 > ############################################### > > Adapter 0: Patrol Read Information: > > Patrol Read Mode: Auto > Patrol Read Execution Delay: 672 hours > Number of iterations completed: 2 > Next start time: 12/13/2014, 03:00:00 > Current State: Stopped > Patrol Read on SSD Devices: Disabled > > Exit Code: 0x00 > ############################################### > > Check Consistency on VD #0 is not in progress. > Check Consistency on VD #1 is not in progress. > Check Consistency on VD #2 is not in progress. > Check Consistency on VD #3 is not in progress. > Check Consistency on VD #4 is not in progress. > Check Consistency on VD #5 is not in progress. > Check Consistency on VD #6 is not in progress. > Check Consistency on VD #7 is not in progress. > Check Consistency on VD #8 is not in progress. > Check Consistency on VD #9 is not in progress. > Check Consistency on VD #10 is not in progress. > Check Consistency on VD #11 is not in progress. > Check Consistency on VD #12 is not in progress. > Check Consistency on VD #13 is not in progress. > Check Consistency on VD #14 is not in progress. > Check Consistency on VD #15 is not in progress. > > Exit Code: 0x00 > root@colossus:~ # ls -l /dev/mfi* > crw-r----- 1 root operator 0x22 Dec 5 17:18 /dev/mfi0 > crw-r----- 1 root operator 0x68 Dec 5 17:18 /dev/mfid0 > crw-r----- 1 root operator 0x69 Dec 5 17:18 /dev/mfid1 > crw-r----- 1 root operator 0x78 Dec 5 17:18 /dev/mfid10 > crw-r----- 1 root operator 0x79 Dec 5 17:18 /dev/mfid11 > crw-r----- 1 root operator 0x7a Dec 5 17:18 /dev/mfid12 > crw-r----- 1 root operator 0x82 Dec 5 17:18 /dev/mfid13 > crw-r----- 1 root operator 0x83 Dec 5 17:18 /dev/mfid14 > crw-r----- 1 root operator 0x84 Dec 5 17:18 /dev/mfid15 > crw-r----- 1 root operator 0x6a Dec 5 17:18 /dev/mfid2 > crw-r----- 1 root operator 0x6b Dec 5 17:18 /dev/mfid3 > crw-r----- 1 root operator 0x6c Dec 5 17:18 /dev/mfid4 > crw-r----- 1 root operator 0x6d Dec 5 17:18 /dev/mfid5 > crw-r----- 1 root operator 0x6e Dec 5 17:18 /dev/mfid6 > crw-r----- 1 root operator 0x75 Dec 5 17:18 /dev/mfid7 > crw-r----- 1 root operator 0x76 Dec 5 17:18 /dev/mfid8 > crw-r----- 1 root operator 0x77 Dec 5 17:18 /dev/mfid9 > root@colossus:~ # >