From owner-freebsd-scsi@FreeBSD.ORG Wed Mar 7 17:45:56 2012 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 30BEF106566B for ; Wed, 7 Mar 2012 17:45:56 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog107.obsmtp.com (na3sys009aog107.obsmtp.com [74.125.149.197]) by mx1.freebsd.org (Postfix) with ESMTP id 7E4DB8FC12 for ; Wed, 7 Mar 2012 17:45:55 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob107.postini.com ([74.125.148.12]) with SMTP ID DSNKT1eezFgpn+kOWJbYq61N5Lhvo8RQJDLu@postini.com; Wed, 07 Mar 2012 09:45:55 PST Received: from PALCAS01.lsi.com (128.94.213.117) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.213.0; Wed, 7 Mar 2012 12:49:54 -0500 Received: from inbexch02.lsi.com (135.36.98.40) by PALCAS01.lsi.com (128.94.213.117) with Microsoft SMTP Server (TLS) id 8.3.213.0; Wed, 7 Mar 2012 12:44:37 -0500 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch02.lsi.com ([135.36.98.40]) with mapi; Wed, 7 Mar 2012 23:14:33 +0530 From: "Desai, Kashyap" To: "dgilbert@interlog.com" , Jason Wolfe Date: Wed, 7 Mar 2012 23:14:29 +0530 Thread-Topic: LSI2008 controller clobbers first disk with new LSI mps driver Thread-Index: Acz1qVrkiusUZ1ojROe2toFMjfTklwBVq20wAWJDjWA= Message-ID: References: <4F450814.4020100@interlog.com> <4F4C14A8.3050105@interlog.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-scsi@freebsd.org" , "McConnell, Stephen" Subject: RE: LSI2008 controller clobbers first disk with new LSI mps driver X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2012 17:45:56 -0000 Jason, We discuss this issue with our architect and he has strong recommendation n= ot to provide any work-around where Enclosure configuration is not correct. Similar issue was reported by other customer sometimes back and they have a= lso configured their Enclosure to resolve this issue. "The enclosure configuration needs to be fixed so it advertises enough slot= s (phys disks + num of SES devices) and it places the SES devices (assigned= slot numbers) above the physical disks." ` Kashyap > -----Original Message----- > From: Desai, Kashyap > Sent: Wednesday, February 29, 2012 10:08 PM > To: 'dgilbert@interlog.com'; Jason Wolfe > Cc: freebsd-scsi@freebsd.org; McConnell, Stephen > Subject: RE: LSI2008 controller clobbers first disk with new LSI mps > driver >=20 > Hi Jason, >=20 > I have started discussion with LSI internal folks to get better clarity > on this issue. Since our key person is on vacation, we may get clarity > on this next week. > I cannot provide some temporary workaround in upstream(because this is > against our design), but if you want to use for your environment, I can > provide you some temporary patch. >=20 > Doug, >=20 > Thanks for providing your view and I have convey this to our architect. >=20 > ~ Kashyap >=20 > > -----Original Message----- > > From: Douglas Gilbert [mailto:dgilbert@interlog.com] > > Sent: Tuesday, February 28, 2012 5:11 AM > > To: Jason Wolfe > > Cc: Desai, Kashyap; freebsd-scsi@freebsd.org; McConnell, Stephen > > Subject: Re: LSI2008 controller clobbers first disk with new LSI mps > > driver > > > > On 12-02-27 02:59 PM, Jason Wolfe wrote: > > > On Wed, Feb 22, 2012 at 9:11 AM, Desai, > Kashyap > > wrote: > > >> > > >> > > >>> -----Original Message----- > > >>> From: Douglas Gilbert [mailto:dgilbert@interlog.com] > > >>> Sent: Wednesday, February 22, 2012 8:52 PM > > >>> To: Desai, Kashyap > > >>> Cc: Jason Wolfe; freebsd-scsi@freebsd.org; McConnell, Stephen > > >>> Subject: Re: LSI2008 controller clobbers first disk with new LSI > mps > > >>> driver > > >>> > > >>> On 12-02-22 03:39 AM, Desai, Kashyap wrote: > > >>>> Here is a possible root cause of this issue. > > >>>> > > >>>> Enclosure which you are using in your setup (might be) not > > configured > > >>> properly. > > >>>> > > >>>> You have Enclosure with 12 Slots + 1 SES Device. > > >>>> See below detail from the log. > > >>>> > > >>>> EventDataLength: 5 > > >>>> AckRequired: 0 > > >>>> Event: SasEnclDeviceStatusChange (0x1d) > > >>>> EventContext: 0x0 > > >>>> EnclosureHandle: 0x2 > > >>>> ReasonCode: Added > > >>>> PhysicalPort: 0 > > >>>> NumSlots: 13 > > >>>> StartSlot: 0 > > >>>> PhyBits: 0xff > > >>>> > > >>>> StartSlot is 0 in this case. > > >>>> Correct behavior should be each device on your enclosure must > have > > >>> different slot number starting from 0 till 12. > > >>>> I have doubt that SES device has not configured well and it is > > using > > >>> slot-0 as default. This can create issue for actual device which > is > > >>> connected to slot-0. > > >>>> So In your setup you will have slot-0 till slot-11 assigned for > > actual > > >>> Phys of your enclosures and again slot-0 is assigned for SES > device > > >>> instead of Slot-12. > > >>> > > >>> No. SAS-2 expanders typically have an integral SES device on an > > >>> expander _virtual_ phy (see SMP DISCOVER (LIST) response). Once > > >>> you see that virtual phy flag the slot number is irrelevant. > > >> > > >> Doug, > > >> > > >> I need some more info so that I can understand your point better. > > >> > > >> I have one Enclosure setup on FreeBSD. Here is smp_discover output. > > (smp_discover_list is failing for me) > > >> > > >> phy 0: inaccessible (phy vacant) > > >> phy 1: inaccessible (phy vacant) > > >> phy 2: inaccessible (phy vacant) > > >> phy 3: inaccessible (phy vacant) > > >> phy 4:S:attached:[500605b012345888:03 i(SSP+STP+SMP)] 6 Gbps > > >> phy 5:S:attached:[500605b012345888:02 i(SSP+STP+SMP)] 6 Gbps > > >> phy 6:S:attached:[500605b012345888:01 i(SSP+STP+SMP)] 6 Gbps > > >> phy 7:S:attached:[500605b012345888:00 i(SSP+STP+SMP)] 6 Gbps > > >> phy 12:D:attached:[5000c5003bc2c389:00 t(SSP)] 6 Gbps > > >> phy 13:D:attached:[500000e116ee91e2:00 t(SSP)] 6 Gbps > > >> phy 14:D:attached:[5000c5003bc308e5:00 t(SSP)] 6 Gbps > > >> phy 15:D:attached:[5000c5003bc2f0d1:00 t(SSP)] 6 Gbps > > >> phy 16:D:attached:[5000c5003bc2ff3d:00 t(SSP)] 6 Gbps > > >> phy 17:D:attached:[5000c5003bae5fdd:00 t(SSP)] 6 Gbps > > >> phy 18:D:attached:[5000c5003bae5eb1:00 t(SSP)] 6 Gbps > > >> phy 19:D:attached:[5000c5003bc2d135:00 t(SSP)] 6 Gbps > > >> phy 20:D:attached:[5000c5003baea36d:00 t(SSP)] 6 Gbps > > >> phy 21:D:attached:[5000c5003bc2a8c9:00 t(SSP)] 6 Gbps > > >> phy 22:D:attached:[5000c5003bc237a9:00 t(SSP)] 6 Gbps > > >> phy 23:D:attached:[5000c5003bc2cec1:00 t(SSP)] 6 Gbps > > >> phy 24:D:attached:[500000e01d92cb52:00 t(SSP)] 3 Gbps > > >> phy 25:D:attached:[500000e01d74cfb2:00 t(SSP)] 3 Gbps > > >> phy 26:D:attached:[500000e01d656052:00 t(SSP)] 3 Gbps > > >> phy 27:D:attached:[500000e01d7cad52:00 t(SSP)] 3 Gbps > > >> phy 28:D:attached:[500c04f2b64cdd1c:00 t(SATA)] 3 Gbps > > >> phy 29:D:attached:[500c04f2b64cdd1d:00 t(SATA)] 3 Gbps > > >> phy 30:D:attached:[500000e01d73c262:00 t(SSP)] 3 Gbps > > >> phy 31:D:attached:[500000e01d536b22:00 t(SSP)] 3 Gbps > > >> phy 32:D:attached:[500000e01d92cab2:00 t(SSP)] 3 Gbps > > >> phy 33:D:attached:[500000e01afd8792:00 t(SSP)] 3 Gbps > > >> phy 34:D:attached:[5000c5003bc30301:00 t(SSP)] 6 Gbps > > >> phy 35:D:attached:[5000c5003bb09a69:00 t(SSP)] 6 Gbps > > >> phy 36:D:attached:[500c04f2b64cdd3d:00 V i(SSP) t(SSP)] 6 > Gbps<- > > -- This has virtual phy set. > > >> > > >> What I understood from your explanation is if we have virt_phy > field > > set, we should not trust slot for that entry. > > >> You are suggesting to use phy index instead of slot. Just for info: > > But how to see Slot details mapping with phy ? > > > > Kashyap, > > I haven't written a SAS discover algorithm but there > > must be plenty of examples out there. One way to do it > > is to find all the phy_ids attached to targets, in this > > case there are SAS (SSP) and SATA targets. Each SATA > > target phy_id will correspond to one SATA disk (or could be an > > ATAPI device (e.g. DVD/BD player)). The SSP targets are a > > bit trickier because two (or more) phys could be connected > > to the same target (either a wide port or multiple (target) > > ports). With a wide port each component phy has the same > > attached SAS address (so above you have a wide initiator > > port (phy ids 4,5,6,7) but no wide target ports). If a > > SAS disk has multiple target ports connected, FreeBSD > > probably has a device node for each. So for each SCSI (SSP) > > target port you need a REPORT LUNS command issued on LUN 0 > > (or the REPORT LUNS well known logical unit) to find the > > LUs it contains. A device node is created for each LU. > > > > Anyway I'm sure many folks in LSI know the SAS discover > > process better than I do. Ask them :-) Surely most of > > the above is already done in your HBA's firmware. > > > > > > BTW I don't think slot numbers are reliable and don't apply > > to things on virtual phys so they will just cause you > > problems when used in the discover process, as this thread > > attests. The BIOS on LSI's HBAs does a discover process > > but is only interested in bootable devices so SES devices > > don't appear. > > > > > > Doug Gilbert > > > > > Kashyap, > > > > > > Let me know if there are any changes agreed upon, I'm happy to test > > > out patches as this is affecting a large number of our machines. I > > > can only imagine the same for others as they start to upgrade, as > this > > > is standard SuperMicro hardware. > > > > > > Thanks, > > > Jason > > >