Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 09 Jul 2015 02:13:57 +0200
From:      Michelle Sullivan <michelle@sorbs.net>
To:        "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>
Subject:   Thanks...
Message-ID:  <559DBCC5.3000601@sorbs.net>

next in thread | raw e-mail | index | archive | help
Well I'm thinking someone patched something in ZFS since my last
'trouble'... because replacing a dead drive just worked this time...
(currently using 9.2-GENERIC-p15)... was so smooth I thought it had
failed at first.. last time I ended up with a crashed pool that took 3
weeks to recover...!

So whomever you are, thank you...!!!

Posting the log just so people searching the web can find tips on how it
should work.... 

Replaced a dead drive (slot 2) with a new one...  LSI-9260-16i
controller + zfs.

root@colossus:~ # ./lsi.sh drives
Slot Number: 0 - Online, Spun Up
Slot Number: 1 - Online, Spun Up
Slot Number: 2 - Unconfigured(good), Spun Up
Slot Number: 3 - Online, Spun Up
Slot Number: 4 - Online, Spun Up
Slot Number: 5 - Online, Spun Up
Slot Number: 6 - Online, Spun Up
Slot Number: 7 - Online, Spun Up
Slot Number: 8 - Online, Spun Up
Slot Number: 9 - Online, Spun Up
Slot Number: 10 - Online, Spun Up
Slot Number: 11 - Online, Spun Up
Slot Number: 12 - Online, Spun Up
Slot Number: 13 - Online, Spun Up
Slot Number: 14 - Online, Spun Up
Slot Number: 15 - Online, Spun Up
root@colossus:~ # praid

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU           
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing       

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts        
0                       
-                                                                             


---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed        
0   15                   16              0               0              
0               0             

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level    
0   L0                   mfid0           optimal         2.728 tb       
raid-0        
0   L1                                   optimal         2.728 tb       
raid-0        
0   L10                                  optimal         2.728 tb       
raid-0        
0   L11                                  optimal         2.728 tb       
raid-0        
0   L12                                  optimal         2.728 tb       
raid-0        
0   L13                                  optimal         2.728 tb       
raid-0        
0   L14                                  optimal         2.728 tb       
raid-0        
0   L15                                  optimal         2.728 tb       
raid-0        
0   L3                                   optimal         2.728 tb       
raid-0        
0   L4                                   optimal         2.728 tb       
raid-0        
0   L5                                   optimal         2.728 tb       
raid-0        
0   L6                                   optimal         2.728 tb       
raid-0        
0   L7                                   optimal         2.728 tb       
raid-0        
0   L8                                   optimal         2.728 tb       
raid-0        
0   L9                                   optimal         2.728 tb       
raid-0        

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address     
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               1          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
unconfigured(good),spun_up none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
root@colossus:~ # megacli -PDMakeGood -PhysDrv\[245:2\] -a0
                                    
Adapter: 0: Failed to change PD state at EnclId-245 SlotId-2.

Exit Code: 0x01
root@colossus:~ # megacli -CfgForeign -Clear -aALL -NoLog
                                    
There is no foreign configuration on controller 0.

Exit Code: 0x00
root@colossus:~ # megacli -cfgldadd -r0\[245:2\] WB RA Cached
CachedBadBBU -strpsz512 -a0
                                    
Adapter 0: Created VD 2

Adapter 0: Configured the Adapter!!

Exit Code: 0x00
root@colossus:~ # praid

MegaCli Tools Used to Gather Raid Info

==== Controllers
=======================================================================================================================
C#  Name                 CacheSize       FirmwareVer     BIOSver        
BBU           
0   lsi megaraid sas 9260-16i 512mb           2.130.403-3066  3.30.02.0
4.16.08.00 0x06060900 Missing       

---- BBUs
------------------------------------------------------------------------------------------------------------------------------
C#  Battery Type         Initialized     Voltage         Temperature    
Charge State    Alerts        
0                       
-                                                                             


---- Virtual Disks
---------------------------------------------------------------------------------------------------------------------
C#  Logical Volumes      Physical Disks  Degraded        Offline        
Critical        Failed        
0   16                   16              0               0              
0               0             

---- Virtual Disks Info
----------------------------------------------------------------------------------------------------------------
C#  ID                   Name            State           Size           
Raid Level    
0   L0                   mfid0           optimal         2.728 tb       
raid-0        
0   L1                                   optimal         2.728 tb       
raid-0        
0   L10                                  optimal         2.728 tb       
raid-0        
0   L11                                  optimal         2.728 tb       
raid-0        
0   L12                                  optimal         2.728 tb       
raid-0        
0   L13                                  optimal         2.728 tb       
raid-0        
0   L14                                  optimal         2.728 tb       
raid-0        
0   L15                                  optimal         2.728 tb       
raid-0        
0   L2                                   optimal         2.728 tb       
raid-0        
0   L3                                   optimal         2.728 tb       
raid-0        
0   L4                                   optimal         2.728 tb       
raid-0        
0   L5                                   optimal         2.728 tb       
raid-0        
0   L6                                   optimal         2.728 tb       
raid-0        
0   L7                                   optimal         2.728 tb       
raid-0        
0   L8                                   optimal         2.728 tb       
raid-0        
0   L9                                   optimal         2.728 tb       
raid-0        

---- Controller 0 Physical Drives
------------------------------------------------------------------------------------------------------
Enclosure: 245 (a0 => unavailable)
C#  Virtual Member       Slot   Model                    
Size            Serial          Media Err  Other Err 
Status               F-State  SAS Address     
0                        0      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320010
0                        1      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320011
0                        10     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320022
0                        11     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320023
0                        12     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320018
0                        13     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320019
0                        14     -                        
2.728tb         -               1          0         
online,spun_up       none     0x500062b20032001a
0                        15     -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032001b
0                        2      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320012
0                        3      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320013
0                        4      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000c
0                        5      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000d
0                        6      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000e
0                        7      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b20032000f
0                        8      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320020
0                        9      -                        
2.728tb         -               0          0         
online,spun_up       none     0x500062b200320021
root@colossus:~ # zpool list
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
storage  40.8T  27.0T  13.8T    66%  1.00x  DEGRADED  -
root@colossus:~ # zpool status -x
  pool: storage
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 83h12m with 0 errors on Wed Jul  8 11:12:07 2015
config:

    NAME                        STATE     READ WRITE CKSUM
    storage                     DEGRADED     0     0     0
      raidz2-0                  DEGRADED     0     0     0
        mfid14                  ONLINE       0     0     0
        mfid12                  ONLINE       0     0     0
        spare-2                 DEGRADED     0     0     0
          15820272272734706674  REMOVED      0     0     0  was /dev/mfid0
          mfid15                ONLINE       0     0     0
        mfid1                   ONLINE       0     0     0
        mfid2                   ONLINE       0     0     1
        mfid3                   ONLINE       0     0     0
        mfid4                   ONLINE       0     0     0
        mfid11                  ONLINE       0     0     0
        mfid5                   ONLINE       0     0     0
        mfid13                  ONLINE       0     0     0
        mfid6                   ONLINE       0     0     0
        mfid7                   ONLINE       0     0     0
        mfid8                   ONLINE       0     0     0
        mfid9                   ONLINE       0     0     0
        mfid10                  ONLINE       0     0     0
    spares
      14948854088277424304      INUSE     was /dev/mfid15

errors: No known data errors
root@colossus:~ # ls -l /dev/mfid*
mfid0%  mfid1%  mfid10% mfid11% mfid12% mfid13% mfid14% mfid15% mfid2% 
mfid3%  mfid4%  mfid5%  mfid6%  mfid7%  mfid8%  mfid9% 
root@colossus:~ # zpool replace
missing pool name argument
usage:
    replace [-f] <pool> <device> [new-device]
root@colossus:~ # zpool replace storage 15820272272734706674 mfid0
root@colossus:~ # zpool status -x
  pool: storage
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Jul  9 02:00:48 2015
        15.9M scanned out of 27.0T at 1.99M/s, (scan is slow, no
estimated time)
        980K resilvered, 0.00% done
config:

    NAME                          STATE     READ WRITE CKSUM
    storage                       DEGRADED     0     0     0
      raidz2-0                    DEGRADED     0     0     0
        mfid14                    ONLINE       0     0     0
        mfid12                    ONLINE       0     0     0
        spare-2                   DEGRADED     0     0     0
          replacing-0             REMOVED      0     0     0
            15820272272734706674  REMOVED      0     0     0  was
/dev/mfid0/old
            mfid0                 ONLINE       0     0     0  (resilvering)
          mfid15                  ONLINE       0     0     0
        mfid1                     ONLINE       0     0     0
        mfid2                     ONLINE       0     0     1
        mfid3                     ONLINE       0     0     0
        mfid4                     ONLINE       0     0     0
        mfid11                    ONLINE       0     0     0
        mfid5                     ONLINE       0     0     0
        mfid13                    ONLINE       0     0     0
        mfid6                     ONLINE       0     0     0
        mfid7                     ONLINE       0     0     0
        mfid8                     ONLINE       0     0     0
        mfid9                     ONLINE       0     0     0
        mfid10                    ONLINE       0     0     0
    spares
      14948854088277424304        INUSE     was /dev/mfid15

errors: No known data errors
root@colossus:~ # uname -a
FreeBSD colossus 9.2-RELEASE-p15 FreeBSD 9.2-RELEASE-p15 #0: Mon Nov  3
20:31:29 UTC 2014    
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
root@colossus:~ #

-- 
Michelle Sullivan
http://www.mhix.org/




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?559DBCC5.3000601>