From owner-freebsd-questions@FreeBSD.ORG Wed Apr 17 22:39:08 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F05DC44E for ; Wed, 17 Apr 2013 22:39:08 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id C126CB0B for ; Wed, 17 Apr 2013 22:39:08 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa03.fnfis.com (8.14.5/8.14.5) with ESMTP id r3HMd2KZ030080 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Wed, 17 Apr 2013 17:39:02 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Wed, 17 Apr 2013 17:39:02 -0500 From: "Teske, Devin" To: Outback Dingo Subject: Re: gmultipath, ses and shared disks / cant seem to share between local nodes Thread-Topic: gmultipath, ses and shared disks / cant seem to share between local nodes Thread-Index: AQHOO7qiCxmtUgKg806i8wlzqyn++JjbVRKA Date: Wed, 17 Apr 2013 22:39:01 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F05E0A@ltcfiswmsgmb21> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.121] Content-Type: text/plain; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-04-17_07:2013-04-17,2013-04-17,1970-01-01 signatures=0 Cc: FreeBSD Questions X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2013 22:39:09 -0000 On Apr 17, 2013, at 3:26 PM, Outback Dingo wrote: > Ok, maybe im at a loss here in the way my brain is viewing this >=20 > we have a box, its got 2 nodes in the chassis, and 32 sata drives > attached to a SATA/SAS backplane via 4 (2 per node) LSI MPT SAS2 cards > should i not logically be seeing 4 controllers X #drive count ?? >=20 > camcontrol devlist shows 32 devices, daX,passX and sesX,passX >=20 > at scbus0 target 9 lun 0 (da0,pass0) > at scbus0 target 10 lun 0 (ses0,pass1) > at scbus0 target 11 lun 0 (da1,pass2) > at scbus0 target 12 lun 0 (ses1,pass3) > at scbus0 target 13 lun 0 (da2,pass4) > at scbus0 target 14 lun 0 (ses2,pass5) > at scbus0 target 15 lun 0 (da3,pass6) > at scbus0 target 16 lun 0 (ses3,pass7) > at scbus0 target 17 lun 0 (da4,pass8) > at scbus0 target 18 lun 0 (ses4,pass9) > at scbus0 target 19 lun 0 (da5,pass10) > at scbus0 target 20 lun 0 (ses5,pass11) > at scbus0 target 21 lun 0 (da6,pass12) > at scbus0 target 22 lun 0 (ses6,pass13) > at scbus0 target 23 lun 0 (da7,pass14) > at scbus0 target 24 lun 0 (ses7,pass15) > at scbus1 target 0 lun 0 (da8,pass16) > at scbus1 target 1 lun 0 (da9,pass17) > at scbus8 target 10 lun 0 (ses8,pass19) > at scbus8 target 11 lun 0 (da11,pass20) > at scbus8 target 12 lun 0 (ses9,pass21) > at scbus8 target 13 lun 0 (da12,pass22) > at scbus8 target 14 lun 0 (ses10,pass23) > at scbus8 target 15 lun 0 (da13,pass24) > at scbus8 target 16 lun 0 (ses11,pass25) > at scbus8 target 17 lun 0 (da14,pass26) > at scbus8 target 18 lun 0 (ses12,pass27) > at scbus8 target 19 lun 0 (da15,pass28) > at scbus8 target 20 lun 0 (ses13,pass29) > at scbus8 target 21 lun 0 (da16,pass30) > at scbus8 target 22 lun 0 (ses14,pass31) > at scbus8 target 23 lun 0 (da17,pass32) > at scbus8 target 24 lun 0 (ses15,pass33) > at scbus9 target 0 lun 0 (da18,pass34) >=20 >=20 > we would like to create a zpool from all the devices, that in theory if > nodeA failed > then nodeB could force import the pool, gmultipath (which you mention in the subject) is the appropriate tool for t= his, but there's no need for an import of the pool if you build the pool ou= t of multipath devices. In our experience, we can pull a cable and zfs cont= inues working just fine. In other words, don't build the pool out of the devices, put a gmultipath l= abel on each device and then use /dev/multipath/LABEL for the zpool devices. > nodeA and NodeB are attached through > dual LSI controllers, to the SATA/SAS backplane. but i cant seem to create > a zpool from sesX or passX devices, i can however create a 16 drive zp0ol > on either node, from any daX device. what did i miss? ive looked at > gmirror, and also ses documents. Any insight is appreciated, thanks in > advance. gmirror is the wrong tool, gmultipath is what you want. The basic task is t= o use "gmultipath label FOO da#" to write a cookie on the disk (used to ide= ntify new/existing paths during GOEM "taste" events for example). After you've labeled the da# devices with gmultipath you say "gmultipath st= atus" to see the components of each label and you use "multipath/LABEL" as = your disk name when creating the zpool (these correspond directly to /dev/m= ultipath/LABEL, but "zpool create =85" or "zpool add =85" allow you to omit= the leading "/dev"). --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.