Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 06 Dec 2014 02:40:00 +0100
From:      Michelle Sullivan <michelle@sorbs.net>
To:        freebsd-fs@freebsd.org
Subject:   ZFS weird issue...
Message-ID:  <54825E70.20900@sorbs.net>

next in thread | raw e-mail | index | archive | help
Here's what happened:

LSI9260-16i, 16x3T SATA drives.  15 LDs each configured as single disk
RAID0 last drive a hot spare.

All LDs configured as a ZFS RaidZ2 pool ('sorbs')

Bay/Drive 8 failed, hot spare kicked in.  ZFS resilvered. (mfid15 became
spare-8)

On reboot mfid9-15 were re-named automatically to mfid8-14 ... ZFS
didn't seems to care...

Days later new drive to replace the dead drive arrived and was
inserted.  System refused to re-add as there was data in the cache, so
rebooted and cleared the cache (as per many on web faq's)  Reconfigured
it to match the others.  Can't do a zpool replace mfid8 because that's
already in the pool... (was mfid9) can't use mfid15 because zpool
reports it's not part of the config... can't use the uniq-id it received
(can't find vdev) ... HELP!! :)

This is what I can see currently ...

Thanks, in advance.

root@colossus:~ # zpool status -v
  pool: VirtualDisks
 state: ONLINE
  scan: none requested
config:

    NAME                       STATE     READ WRITE CKSUM
    VirtualDisks               ONLINE       0     0     0
      zvol/sorbs/VirtualDisks  ONLINE       0     0     0

errors: No known data errors

  pool: sorbs
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas
exist for
    the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub in progress since Fri Dec  5 17:11:29 2014
        2.51T scanned out of 29.9T at 89.4M/s, 89h7m to go
        0 repaired, 8.40% done
config:

    NAME              STATE     READ WRITE CKSUM
    sorbs             DEGRADED     0     0     0
      raidz2-0        DEGRADED     0     0     0
        mfid0         ONLINE       0     0     0
        mfid1         ONLINE       0     0     0
        mfid2         ONLINE       0     0     0
        mfid3         ONLINE       0     0     0
        mfid4         ONLINE       0     0     0
        mfid5         ONLINE       0     0     0
        mfid6         ONLINE       0     0     0
        mfid7         ONLINE       0     0     0
        spare-8       DEGRADED     0     0     0
          1702922605  UNAVAIL      0     0     0  was /dev/mfid8
          mfid14      ONLINE       0     0     0
        mfid8         ONLINE       0     0     0
        mfid9         ONLINE       0     0     0
        mfid10        ONLINE       0     0     0
        mfid11        ONLINE       0     0     0
        mfid12        ONLINE       0     0     0
        mfid13        ONLINE       0     0     0
    spares
      933862663       INUSE     was /dev/mfid14

errors: No known data errors
root@colossus:~ # uname -a
FreeBSD colossus.sorbs.net 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898:
Thu Sep 26 22:50:31 UTC 2013    
root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
root@colossus:~ # sh ./lsi.sh drives
Slot Number: 0 - Online, Spun Up
Slot Number: 1 - Online, Spun Up
Slot Number: 2 - Online, Spun Up
Slot Number: 3 - Online, Spun Up
Slot Number: 4 - Online, Spun Up
Slot Number: 5 - Online, Spun Up
Slot Number: 6 - Online, Spun Up
Slot Number: 7 - Online, Spun Up
Slot Number: 8 - Online, Spun Up
Slot Number: 9 - Online, Spun Up
Slot Number: 10 - Online, Spun Up
Slot Number: 11 - Online, Spun Up
Slot Number: 12 - Online, Spun Up
Slot Number: 13 - Online, Spun Up
Slot Number: 14 - Online, Spun Up
Slot Number: 15 - Online, Spun Up
root@colossus:~ # sh ./lsi.sh status
                                    

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 1 (Target Id: 1)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 2 (Target Id: 2)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 3 (Target Id: 3)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 4 (Target Id: 4)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 5 (Target Id: 5)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 6 (Target Id: 6)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 7 (Target Id: 7)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 8 (Target Id: 8)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 512 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 9 (Target Id: 9)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 10 (Target Id: 10)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 11 (Target Id: 11)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 12 (Target Id: 12)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 13 (Target Id: 13)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 14 (Target Id: 14)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 64 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No


Virtual Drive: 15 (Target Id: 15)
Name                :
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Size                : 2.728 TB
Sector Size         : 512
Is VD emulated      : Yes
Parity Size         : 0
State               : Optimal
Strip Size          : 256 KB
Number Of Drives    : 1
Span Depth          : 1
Default Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Cached, Write Cache OK if
Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Enabled
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: No



Exit Code: 0x00
###############################################
                                    
Adapter 0: Patrol Read Information:

Patrol Read Mode: Auto
Patrol Read Execution Delay: 672 hours
Number of iterations completed: 2
Next start time: 12/13/2014, 03:00:00
Current State: Stopped
Patrol Read on SSD Devices: Disabled

Exit Code: 0x00
###############################################
                                    
Check Consistency on VD #0 is not in progress.
Check Consistency on VD #1 is not in progress.
Check Consistency on VD #2 is not in progress.
Check Consistency on VD #3 is not in progress.
Check Consistency on VD #4 is not in progress.
Check Consistency on VD #5 is not in progress.
Check Consistency on VD #6 is not in progress.
Check Consistency on VD #7 is not in progress.
Check Consistency on VD #8 is not in progress.
Check Consistency on VD #9 is not in progress.
Check Consistency on VD #10 is not in progress.
Check Consistency on VD #11 is not in progress.
Check Consistency on VD #12 is not in progress.
Check Consistency on VD #13 is not in progress.
Check Consistency on VD #14 is not in progress.
Check Consistency on VD #15 is not in progress.

Exit Code: 0x00
root@colossus:~ # ls -l /dev/mfi*
crw-r-----  1 root  operator  0x22 Dec  5 17:18 /dev/mfi0
crw-r-----  1 root  operator  0x68 Dec  5 17:18 /dev/mfid0
crw-r-----  1 root  operator  0x69 Dec  5 17:18 /dev/mfid1
crw-r-----  1 root  operator  0x78 Dec  5 17:18 /dev/mfid10
crw-r-----  1 root  operator  0x79 Dec  5 17:18 /dev/mfid11
crw-r-----  1 root  operator  0x7a Dec  5 17:18 /dev/mfid12
crw-r-----  1 root  operator  0x82 Dec  5 17:18 /dev/mfid13
crw-r-----  1 root  operator  0x83 Dec  5 17:18 /dev/mfid14
crw-r-----  1 root  operator  0x84 Dec  5 17:18 /dev/mfid15
crw-r-----  1 root  operator  0x6a Dec  5 17:18 /dev/mfid2
crw-r-----  1 root  operator  0x6b Dec  5 17:18 /dev/mfid3
crw-r-----  1 root  operator  0x6c Dec  5 17:18 /dev/mfid4
crw-r-----  1 root  operator  0x6d Dec  5 17:18 /dev/mfid5
crw-r-----  1 root  operator  0x6e Dec  5 17:18 /dev/mfid6
crw-r-----  1 root  operator  0x75 Dec  5 17:18 /dev/mfid7
crw-r-----  1 root  operator  0x76 Dec  5 17:18 /dev/mfid8
crw-r-----  1 root  operator  0x77 Dec  5 17:18 /dev/mfid9
root@colossus:~ #

-- 
Michelle Sullivan
http://www.mhix.org/




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54825E70.20900>