From owner-freebsd-fs@freebsd.org Thu Jul 9 01:14:06 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17C6299647C for ; Thu, 9 Jul 2015 01:14:06 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (hades.sorbs.net [67.231.146.201]) by mx1.freebsd.org (Postfix) with ESMTP id 094F615D2 for ; Thu, 9 Jul 2015 01:14:05 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0NR700B4X2908X00@hades.sorbs.net> for freebsd-fs@freebsd.org; Wed, 08 Jul 2015 17:19:50 -0700 (PDT) Message-id: <559DBCC5.3000601@sorbs.net> Date: Thu, 09 Jul 2015 02:13:57 +0200 From: Michelle Sullivan User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.24) Gecko/20100301 SeaMonkey/1.1.19 To: "freebsd-fs@freebsd.org" Subject: Thanks... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Jul 2015 01:14:06 -0000 Well I'm thinking someone patched something in ZFS since my last 'trouble'... because replacing a dead drive just worked this time... (currently using 9.2-GENERIC-p15)... was so smooth I thought it had failed at first.. last time I ended up with a crashed pool that took 3 weeks to recover...! So whomever you are, thank you...!!! Posting the log just so people searching the web can find tips on how it should work.... Replaced a dead drive (slot 2) with a new one... LSI-9260-16i controller + zfs. root@colossus:~ # ./lsi.sh drives Slot Number: 0 - Online, Spun Up Slot Number: 1 - Online, Spun Up Slot Number: 2 - Unconfigured(good), Spun Up Slot Number: 3 - Online, Spun Up Slot Number: 4 - Online, Spun Up Slot Number: 5 - Online, Spun Up Slot Number: 6 - Online, Spun Up Slot Number: 7 - Online, Spun Up Slot Number: 8 - Online, Spun Up Slot Number: 9 - Online, Spun Up Slot Number: 10 - Online, Spun Up Slot Number: 11 - Online, Spun Up Slot Number: 12 - Online, Spun Up Slot Number: 13 - Online, Spun Up Slot Number: 14 - Online, Spun Up Slot Number: 15 - Online, Spun Up root@colossus:~ # praid MegaCli Tools Used to Gather Raid Info ==== Controllers ======================================================================================================================= C# Name CacheSize FirmwareVer BIOSver BBU 0 lsi megaraid sas 9260-16i 512mb 2.130.403-3066 3.30.02.0 4.16.08.00 0x06060900 Missing ---- BBUs ------------------------------------------------------------------------------------------------------------------------------ C# Battery Type Initialized Voltage Temperature Charge State Alerts 0 - ---- Virtual Disks --------------------------------------------------------------------------------------------------------------------- C# Logical Volumes Physical Disks Degraded Offline Critical Failed 0 15 16 0 0 0 0 ---- Virtual Disks Info ---------------------------------------------------------------------------------------------------------------- C# ID Name State Size Raid Level 0 L0 mfid0 optimal 2.728 tb raid-0 0 L1 optimal 2.728 tb raid-0 0 L10 optimal 2.728 tb raid-0 0 L11 optimal 2.728 tb raid-0 0 L12 optimal 2.728 tb raid-0 0 L13 optimal 2.728 tb raid-0 0 L14 optimal 2.728 tb raid-0 0 L15 optimal 2.728 tb raid-0 0 L3 optimal 2.728 tb raid-0 0 L4 optimal 2.728 tb raid-0 0 L5 optimal 2.728 tb raid-0 0 L6 optimal 2.728 tb raid-0 0 L7 optimal 2.728 tb raid-0 0 L8 optimal 2.728 tb raid-0 0 L9 optimal 2.728 tb raid-0 ---- Controller 0 Physical Drives ------------------------------------------------------------------------------------------------------ Enclosure: 245 (a0 => unavailable) C# Virtual Member Slot Model Size Serial Media Err Other Err Status F-State SAS Address 0 0 - 2.728tb - 0 0 online,spun_up none 0x500062b200320010 0 1 - 2.728tb - 0 0 online,spun_up none 0x500062b200320011 0 10 - 2.728tb - 0 0 online,spun_up none 0x500062b200320022 0 11 - 2.728tb - 0 0 online,spun_up none 0x500062b200320023 0 12 - 2.728tb - 0 0 online,spun_up none 0x500062b200320018 0 13 - 2.728tb - 0 0 online,spun_up none 0x500062b200320019 0 14 - 2.728tb - 1 0 online,spun_up none 0x500062b20032001a 0 15 - 2.728tb - 0 0 online,spun_up none 0x500062b20032001b 0 2 - 2.728tb - 0 0 unconfigured(good),spun_up none 0x500062b200320012 0 3 - 2.728tb - 0 0 online,spun_up none 0x500062b200320013 0 4 - 2.728tb - 0 0 online,spun_up none 0x500062b20032000c 0 5 - 2.728tb - 0 0 online,spun_up none 0x500062b20032000d 0 6 - 2.728tb - 0 0 online,spun_up none 0x500062b20032000e 0 7 - 2.728tb - 0 0 online,spun_up none 0x500062b20032000f 0 8 - 2.728tb - 0 0 online,spun_up none 0x500062b200320020 0 9 - 2.728tb - 0 0 online,spun_up none 0x500062b200320021 root@colossus:~ # megacli -PDMakeGood -PhysDrv\[245:2\] -a0 Adapter: 0: Failed to change PD state at EnclId-245 SlotId-2. Exit Code: 0x01 root@colossus:~ # megacli -CfgForeign -Clear -aALL -NoLog There is no foreign configuration on controller 0. Exit Code: 0x00 root@colossus:~ # megacli -cfgldadd -r0\[245:2\] WB RA Cached CachedBadBBU -strpsz512 -a0 Adapter 0: Created VD 2 Adapter 0: Configured the Adapter!! Exit Code: 0x00 root@colossus:~ # praid MegaCli Tools Used to Gather Raid Info ==== Controllers ======================================================================================================================= C# Name CacheSize FirmwareVer BIOSver BBU 0 lsi megaraid sas 9260-16i 512mb 2.130.403-3066 3.30.02.0 4.16.08.00 0x06060900 Missing ---- BBUs ------------------------------------------------------------------------------------------------------------------------------ C# Battery Type Initialized Voltage Temperature Charge State Alerts 0 - ---- Virtual Disks --------------------------------------------------------------------------------------------------------------------- C# Logical Volumes Physical Disks Degraded Offline Critical Failed 0 16 16 0 0 0 0 ---- Virtual Disks Info ---------------------------------------------------------------------------------------------------------------- C# ID Name State Size Raid Level 0 L0 mfid0 optimal 2.728 tb raid-0 0 L1 optimal 2.728 tb raid-0 0 L10 optimal 2.728 tb raid-0 0 L11 optimal 2.728 tb raid-0 0 L12 optimal 2.728 tb raid-0 0 L13 optimal 2.728 tb raid-0 0 L14 optimal 2.728 tb raid-0 0 L15 optimal 2.728 tb raid-0 0 L2 optimal 2.728 tb raid-0 0 L3 optimal 2.728 tb raid-0 0 L4 optimal 2.728 tb raid-0 0 L5 optimal 2.728 tb raid-0 0 L6 optimal 2.728 tb raid-0 0 L7 optimal 2.728 tb raid-0 0 L8 optimal 2.728 tb raid-0 0 L9 optimal 2.728 tb raid-0 ---- Controller 0 Physical Drives ------------------------------------------------------------------------------------------------------ Enclosure: 245 (a0 => unavailable) C# Virtual Member Slot Model Size Serial Media Err Other Err Status F-State SAS Address 0 0 - 2.728tb - 0 0 online,spun_up none 0x500062b200320010 0 1 - 2.728tb - 0 0 online,spun_up none 0x500062b200320011 0 10 - 2.728tb - 0 0 online,spun_up none 0x500062b200320022 0 11 - 2.728tb - 0 0 online,spun_up none 0x500062b200320023 0 12 - 2.728tb - 0 0 online,spun_up none 0x500062b200320018 0 13 - 2.728tb - 0 0 online,spun_up none 0x500062b200320019 0 14 - 2.728tb - 1 0 online,spun_up none 0x500062b20032001a 0 15 - 2.728tb - 0 0 online,spun_up none 0x500062b20032001b 0 2 - 2.728tb - 0 0 online,spun_up none 0x500062b200320012 0 3 - 2.728tb - 0 0 online,spun_up none 0x500062b200320013 0 4 - 2.728tb - 0 0 online,spun_up none 0x500062b20032000c 0 5 - 2.728tb - 0 0 online,spun_up none 0x500062b20032000d 0 6 - 2.728tb - 0 0 online,spun_up none 0x500062b20032000e 0 7 - 2.728tb - 0 0 online,spun_up none 0x500062b20032000f 0 8 - 2.728tb - 0 0 online,spun_up none 0x500062b200320020 0 9 - 2.728tb - 0 0 online,spun_up none 0x500062b200320021 root@colossus:~ # zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT storage 40.8T 27.0T 13.8T 66% 1.00x DEGRADED - root@colossus:~ # zpool status -x pool: storage state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://illumos.org/msg/ZFS-8000-9P scan: scrub repaired 0 in 83h12m with 0 errors on Wed Jul 8 11:12:07 2015 config: NAME STATE READ WRITE CKSUM storage DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 mfid14 ONLINE 0 0 0 mfid12 ONLINE 0 0 0 spare-2 DEGRADED 0 0 0 15820272272734706674 REMOVED 0 0 0 was /dev/mfid0 mfid15 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 mfid2 ONLINE 0 0 1 mfid3 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid11 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 spares 14948854088277424304 INUSE was /dev/mfid15 errors: No known data errors root@colossus:~ # ls -l /dev/mfid* mfid0% mfid1% mfid10% mfid11% mfid12% mfid13% mfid14% mfid15% mfid2% mfid3% mfid4% mfid5% mfid6% mfid7% mfid8% mfid9% root@colossus:~ # zpool replace missing pool name argument usage: replace [-f] [new-device] root@colossus:~ # zpool replace storage 15820272272734706674 mfid0 root@colossus:~ # zpool status -x pool: storage state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Thu Jul 9 02:00:48 2015 15.9M scanned out of 27.0T at 1.99M/s, (scan is slow, no estimated time) 980K resilvered, 0.00% done config: NAME STATE READ WRITE CKSUM storage DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 mfid14 ONLINE 0 0 0 mfid12 ONLINE 0 0 0 spare-2 DEGRADED 0 0 0 replacing-0 REMOVED 0 0 0 15820272272734706674 REMOVED 0 0 0 was /dev/mfid0/old mfid0 ONLINE 0 0 0 (resilvering) mfid15 ONLINE 0 0 0 mfid1 ONLINE 0 0 0 mfid2 ONLINE 0 0 1 mfid3 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid11 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid13 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 spares 14948854088277424304 INUSE was /dev/mfid15 errors: No known data errors root@colossus:~ # uname -a FreeBSD colossus 9.2-RELEASE-p15 FreeBSD 9.2-RELEASE-p15 #0: Mon Nov 3 20:31:29 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 root@colossus:~ # -- Michelle Sullivan http://www.mhix.org/