Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 17 Apr 2013 19:10:37 -0400
From:      Outback Dingo <outbackdingo@gmail.com>
To:        Devin Teske <dteske@freebsd.org>
Cc:        FreeBSD Questions <freebsd-questions@freebsd.org>
Subject:   Re: gmultipath, ses and shared disks / cant seem to share between local nodes
Message-ID:  <CAKYr3zw8eK3kHExKynvfngewa7%2BtzxyxPKo5N%2B4tzuoA%2Bz-YbA@mail.gmail.com>
In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F05E0A@ltcfiswmsgmb21>
References:  <CAKYr3zy7fXriB_rD6XuvsZT%2B19JmTAX-882f=MNjz=uhGHwFjA@mail.gmail.com> <13CA24D6AB415D428143D44749F57D7201F05E0A@ltcfiswmsgmb21>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Apr 17, 2013 at 6:39 PM, Teske, Devin <Devin.Teske@fisglobal.com>wr=
ote:

>
> On Apr 17, 2013, at 3:26 PM, Outback Dingo wrote:
>
> > Ok, maybe im at a loss here in the way my brain is viewing this
> >
> > we have a box, its got 2 nodes in the chassis, and 32 sata drives
> > attached to a SATA/SAS backplane via 4 (2 per node) LSI MPT SAS2 cards
> > should i not logically be seeing 4 controllers X #drive count ??
> >
> > camcontrol devlist shows 32 devices, daX,passX and sesX,passX
> >
> > <SEAGATE ST33000650SS 0004>        at scbus0 target 9 lun 0 (da0,pass0)
> > <STORBRICK-3 1400>        at scbus0 target 10 lun 0 (ses0,pass1)
> > <SEAGATE ST33000650SS 0004>        at scbus0 target 11 lun 0 (da1,pass2=
)
> > <STORBRICK-1 1400>        at scbus0 target 12 lun 0 (ses1,pass3)
> > <SEAGATE ST33000650SS 0004>        at scbus0 target 13 lun 0 (da2,pass4=
)
> > <STORBRICK-2 1400>        at scbus0 target 14 lun 0 (ses2,pass5)
> > <SEAGATE ST33000650SS 0004>        at scbus0 target 15 lun 0 (da3,pass6=
)
> > <STORBRICK-4 1400>        at scbus0 target 16 lun 0 (ses3,pass7)
> > <SEAGATE ST33000650SS 0004>        at scbus0 target 17 lun 0 (da4,pass8=
)
> > <STORBRICK-6 1400>        at scbus0 target 18 lun 0 (ses4,pass9)
> > <SEAGATE ST33000650SS 0004>        at scbus0 target 19 lun 0 (da5,pass1=
0)
> > <STORBRICK-0 1400>        at scbus0 target 20 lun 0 (ses5,pass11)
> > <SEAGATE ST33000650SS 0004>        at scbus0 target 21 lun 0 (da6,pass1=
2)
> > <STORBRICK-7 1400>        at scbus0 target 22 lun 0 (ses6,pass13)
> > <SEAGATE ST33000650SS 0004>        at scbus0 target 23 lun 0 (da7,pass1=
4)
> > <STORBRICK-5 1400>        at scbus0 target 24 lun 0 (ses7,pass15)
> > <SEAGATE ST9300605SS 0004>         at scbus1 target 0 lun 0 (da8,pass16=
)
> > <SEAGATE ST9300605SS 0004>         at scbus1 target 1 lun 0 (da9,pass17=
)
> > <STORBRICK-3 1400>        at scbus8 target 10 lun 0 (ses8,pass19)
> > <SEAGATE ST33000650SS 0004>        at scbus8 target 11 lun 0
> (da11,pass20)
> > <STORBRICK-1 1400>        at scbus8 target 12 lun 0 (ses9,pass21)
> > <SEAGATE ST33000650SS 0004>        at scbus8 target 13 lun 0
> (da12,pass22)
> > <STORBRICK-2 1400>        at scbus8 target 14 lun 0 (ses10,pass23)
> > <SEAGATE ST33000650SS 0004>        at scbus8 target 15 lun 0
> (da13,pass24)
> > <STORBRICK-4 1400>        at scbus8 target 16 lun 0 (ses11,pass25)
> > <SEAGATE ST33000650SS 0004>        at scbus8 target 17 lun 0
> (da14,pass26)
> > <STORBRICK-6 1400>        at scbus8 target 18 lun 0 (ses12,pass27)
> > <SEAGATE ST33000650SS 0004>        at scbus8 target 19 lun 0
> (da15,pass28)
> > <STORBRICK-0 1400>        at scbus8 target 20 lun 0 (ses13,pass29)
> > <SEAGATE ST33000650SS 0004>        at scbus8 target 21 lun 0
> (da16,pass30)
> > <STORBRICK-7 1400>        at scbus8 target 22 lun 0 (ses14,pass31)
> > <SEAGATE ST33000650SS 0004>        at scbus8 target 23 lun 0
> (da17,pass32)
> > <STORBRICK-5 1400>        at scbus8 target 24 lun 0 (ses15,pass33)
> > <USB 2.0 Flash Drive 8.07>         at scbus9 target 0 lun 0 (da18,pass3=
4)
> >
> >
> > we would like to create a zpool from all the devices, that in theory if
> > nodeA failed
> > then nodeB could force import the pool,
>
> gmultipath (which you mention in the subject) is the appropriate tool for
> this, but there's no need for an import of the pool if you build the pool
> out of multipath devices. In our experience, we can pull a cable and zfs
> continues working just fine.
>
> In other words, don't build the pool out of the devices, put a gmultipath
> label on each device and then use /dev/multipath/LABEL for the zpool
> devices.
>
>
> > nodeA and NodeB are attached through
> > dual LSI controllers, to the SATA/SAS backplane. but i cant seem to
> create
> > a zpool from sesX or passX devices, i can however create a 16 drive zp0=
ol
> > on either node, from any daX device. what did i miss? ive looked at
> > gmirror, and also ses documents. Any insight is appreciated, thanks in
> > advance.
>
> gmirror is the wrong tool, gmultipath is what you want. The basic task is
> to use "gmultipath label FOO da#" to write a cookie on the disk (used to
> identify new/existing paths during GOEM "taste" events for example).
>
> After you've labeled the da# devices with gmultipath you say "gmultipath
> status" to see the components of each label and you use "multipath/LABEL"
> as your disk name when creating the zpool (these correspond directly to
> /dev/multipath/LABEL, but "zpool create =85" or "zpool add =85" allow you=
 to
> omit the leading "/dev").
>

sanity check me on node A i did

zpool destroy master

gmultipath label FOO da0

gmultipath status
                    Name    Status  Components
           multipath/FOO  DEGRADED  da0 (ACTIVE)
 multipath/FOO-619648737  DEGRADED  da1 (ACTIVE)
 multipath/FOO-191725652  DEGRADED  da2 (ACTIVE)
multipath/FOO-1539342315  DEGRADED  da3 (ACTIVE)
multipath/FOO-1276041606  DEGRADED  da4 (ACTIVE)
multipath/FOO-2000832198  DEGRADED  da5 (ACTIVE)
multipath/FOO-1285640577  DEGRADED  da6 (ACTIVE)
multipath/FOO-1816092574  DEGRADED  da7 (ACTIVE)
multipath/FOO-1102254444  DEGRADED  da8 (ACTIVE)
 multipath/FOO-330300690  DEGRADED  da9 (ACTIVE)
  multipath/FOO-92140635  DEGRADED  da10 (ACTIVE)
 multipath/FOO-855257672  DEGRADED  da11 (ACTIVE)
multipath/FOO-1003634134  DEGRADED  da12 (ACTIVE)
   multipath/FOO-2449862  DEGRADED  da13 (ACTIVE)
multipath/FOO-1137080233  DEGRADED  da14 (ACTIVE)
multipath/FOO-1696804371  DEGRADED  da15 (ACTIVE)
multipath/FOO-1304457562  DEGRADED  da16 (ACTIVE)
 multipath/FOO-912159854  DEGRADED  da17 (ACTIVE)

now on node B i should do the same? reboot both nodes and i should be able
"see" 32 multipath/FOO deices to create a pool from ?




> --
> Devin
>
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete t=
he
> message and all copies; (ii) do not disclose, distribute or use the messa=
ge
> in any manner; and (iii) notify the sender immediately. In addition, plea=
se
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKYr3zw8eK3kHExKynvfngewa7%2BtzxyxPKo5N%2B4tzuoA%2Bz-YbA>