From owner-freebsd-scsi@freebsd.org Thu Feb 25 19:45:55 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35F63AB40FB for ; Thu, 25 Feb 2016 19:45:55 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id 216AB1DAE for ; Thu, 25 Feb 2016 19:45:54 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 25 Feb 2016 12:00:35 -0800 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.9/8.14.4) with ESMTP id u1PJjs6l012775; Thu, 25 Feb 2016 11:45:54 -0800 (PST) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.9/8.14.4/Submit) id u1PJjrbe012773; Thu, 25 Feb 2016 11:45:53 -0800 (PST) (envelope-from ambrisko) Date: Thu, 25 Feb 2016 11:45:53 -0800 From: Doug Ambrisko To: Tinker Cc: freebsd-scsi@freebsd.org Subject: Re: MRSAS driver/LSI MegaRaid 92XX-93XX admin question: When one of the Raid's physical drives break, how is it reported in the logs? Message-ID: <20160225194553.GA10162@ambrisko.com> References: <6a648d421b6d611b4f6f411b66303017@openmailbox.org> <55de137d1ed81930cfdbee579d881d62@openmailbox.org> <20160217000002.GA81916@ambrisko.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160217000002.GA81916@ambrisko.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Feb 2016 19:45:55 -0000 On Tue, Feb 16, 2016 at 04:00:02PM -0800, Doug Ambrisko wrote: | On Sun, Feb 14, 2016 at 10:13:31PM +0700, Tinker wrote: | | (Will send any followup from now only to freebsd-scsi@ .) | | | | Did some additional research and found that the disk failure indeed is | | reported in MRSAS' "event log". | | | | So my final question then is, how do you extract it into userland (in | | the absence of an "mfiutil" as the MFI driver has)? | | I have local changes to print the event log in dmesg which gets sysloged. | We then watch syslog for issues to report things to our customers | automatically. This is similar to mfi(4). I put up a couple of patches: https://people.freebsd.org/~ambrisko/mrsas.patch https://people.freebsd.org/~ambrisko/mrsasutil.patch I made a bunch of changes to the driver to deal with issues we've seen at work. I've done light testing and it is working better now. Most of my testing is under FreeBSD 9.2 but the code base is from -current. It is going through more product testing that exposed issues with the ioctl path. One of the major changes in the ioctl path is let the OS create the SG list since user-land doesn't really know what the kernel memory is like and lets the OS figure it out. It also uses 64 bit address range. Limiting the driver address range was creating problems when the system memory was being used and potentially fragmented resulting in lack of memory that could be allocated. This occurred after our appliance was up for a while and during tests. It also adds support for displaying event logs to dmesg such as: mrsas0: 19366 (509744360s/0x0002/info) - State change on PD 16(e0x00/s1) from ONLINE(18) to FAILED(11) mrsas0: 19367 (509744360s/0x0001/info) - State change on VD 00/0 from OPTIMAL(3) to DEGRADED(2) mrsas0: 19368 (509744360s/0x0001/CRIT) - VD 00/0 is now DEGRADED mrsas0: 19369 (509744371s/0x0002/info) - Rebuild started on PD 16(e0x00/s1) mrsas0: 19370 (509744371s/0x0002/info) - State change on PD 16(e0x00/s1) from FAILED(11) to REBUILD(14) It only happens at run time not at boot like mfi. I also added support for mfiutil and created a patch against mfiutil to create hard link to mrsasutil so it will know to automatically use mrsas0. I created the above logs via mrsasutil fail mrsasutil rebuild Again this is lightly tested. I need to test 32 bit emulation and 32 bit build. I need to test it with current. It's a work in progress. Thanks, Doug A. From owner-freebsd-scsi@freebsd.org Fri Feb 26 12:44:08 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1FB7AB4ED3 for ; Fri, 26 Feb 2016 12:44:08 +0000 (UTC) (envelope-from cristobal@dielmo.com) Received: from mbob.nabble.com (mbob.nabble.com [162.253.133.15]) by mx1.freebsd.org (Postfix) with ESMTP id AE85F1772 for ; Fri, 26 Feb 2016 12:44:07 +0000 (UTC) (envelope-from cristobal@dielmo.com) Received: from msam.nabble.com (unknown [162.253.133.85]) by mbob.nabble.com (Postfix) with ESMTP id C7B6B219EBBA for ; Fri, 26 Feb 2016 04:36:29 -0800 (PST) Date: Fri, 26 Feb 2016 05:44:07 -0700 (MST) From: crigariba To: freebsd-scsi@freebsd.org Message-ID: <1456490647564-6079717.post@n5.nabble.com> In-Reply-To: <548B4C25.8010501@crystal.harvard.edu> References: <54822835.3080800@crystal.harvard.edu> <548B4C25.8010501@crystal.harvard.edu> Subject: Re: LSI SAS 3008 card - 35 out of 36 disks detected MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 12:44:08 -0000 We have a Supermicro chassis with a LSI 3008 HBA card. Installed FreeNas 9.3-RELEASE, but realized there were only 35 disks visible to the OS. The missing disk is in bay 24 (starting at 0 - 1st disk on rear backplane). Both in the card bios and booting a live gnu/linux shows all 36 disks and said disk is functional. Some research on the web suggest using the lsiutil to fix the problem, but we can't find the FreeNas version of the lsiutil. Please, we need help about how can we install and run the lsiutil version for FreeNas. -- View this message in context: http://freebsd.1045724.n5.nabble.com/LSI-SAS-3008-card-35-out-of-36-disks-detected-tp5971350p6079717.html Sent from the freebsd-scsi mailing list archive at Nabble.com. From owner-freebsd-scsi@freebsd.org Fri Feb 26 12:58:10 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7275AAB5624 for ; Fri, 26 Feb 2016 12:58:10 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 31B7E226 for ; Fri, 26 Feb 2016 12:58:09 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.96] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id E13BA9DCA79; Fri, 26 Feb 2016 13:51:57 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: LSI SAS 3008 card - 35 out of 36 disks detected From: Borja Marcos In-Reply-To: <1456490647564-6079717.post@n5.nabble.com> Date: Fri, 26 Feb 2016 13:51:57 +0100 Cc: freebsd-scsi@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <54822835.3080800@crystal.harvard.edu> <548B4C25.8010501@crystal.harvard.edu> <1456490647564-6079717.post@n5.nabble.com> To: crigariba X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 12:58:10 -0000 > On 26 Feb 2016, at 13:44, crigariba wrote: >=20 > Please, we need help about how can we install and run the lsiutil = version > for FreeNas. FreeNAS includes both sas[23]flash and sas[23]ircu, for the LSI2008 and = LSI3008 controllers. Maybe sas3ircu can help? At least it can show you the attached disks and = it can create/delete volumes, etc. Borja. From owner-freebsd-scsi@freebsd.org Fri Feb 26 14:04:31 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 514EDAB5154 for ; Fri, 26 Feb 2016 14:04:31 +0000 (UTC) (envelope-from cristobal@dielmo.com) Received: from mbob.nabble.com (mbob.nabble.com [162.253.133.15]) by mx1.freebsd.org (Postfix) with ESMTP id 3EEF58F5 for ; Fri, 26 Feb 2016 14:04:30 +0000 (UTC) (envelope-from cristobal@dielmo.com) Received: from msam.nabble.com (unknown [162.253.133.85]) by mbob.nabble.com (Postfix) with ESMTP id 0BC9D21A00B2 for ; Fri, 26 Feb 2016 05:56:52 -0800 (PST) Date: Fri, 26 Feb 2016 07:04:30 -0700 (MST) From: crigariba To: freebsd-scsi@freebsd.org Message-ID: <1456495470171-6079739.post@n5.nabble.com> In-Reply-To: References: <54822835.3080800@crystal.harvard.edu> <548B4C25.8010501@crystal.harvard.edu> <1456490647564-6079717.post@n5.nabble.com> Subject: Re: LSI SAS 3008 card - 35 out of 36 disks detected MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 14:04:31 -0000 Hello, Borja. sas2ircu returns me "No controllers Found" xxxx@freenas:~ % sas2ircu list LSI Corporation SAS2 IR Configuration Utility. Version 20.00.00.00 (2014.09.18) Copyright (c) 2008-2014 LSI Corporation. All rights reserved. SAS2IRCU: No Controllers Found. -- View this message in context: http://freebsd.1045724.n5.nabble.com/LSI-SAS-3008-card-35-out-of-36-disks-detected-tp5971350p6079739.html Sent from the freebsd-scsi mailing list archive at Nabble.com. From owner-freebsd-scsi@freebsd.org Fri Feb 26 15:53:49 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BFF23AB5CD4 for ; Fri, 26 Feb 2016 15:53:49 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from mail.physics.umn.edu (smtp.spa.umn.edu [128.101.220.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9F5E7F05 for ; Fri, 26 Feb 2016 15:53:49 +0000 (UTC) (envelope-from allan@physics.umn.edu) Received: from c-66-41-25-68.hsd1.mn.comcast.net ([66.41.25.68] helo=[192.168.0.102]) by mail.physics.umn.edu with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1aZKOh-00098w-GQ for freebsd-scsi@freebsd.org; Fri, 26 Feb 2016 09:33:39 -0600 Subject: Re: LSI SAS 3008 card - 35 out of 36 disks detected To: freebsd-scsi@freebsd.org References: <54822835.3080800@crystal.harvard.edu> <548B4C25.8010501@crystal.harvard.edu> <1456490647564-6079717.post@n5.nabble.com> <20160226152657.GA75063@neutralgood.org> From: Graham Allan Message-ID: <56D0704D.1080906@physics.umn.edu> Date: Fri, 26 Feb 2016 09:33:33 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <20160226152657.GA75063@neutralgood.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 15:53:49 -0000 On 2/26/2016 9:26 AM, kpneal@pobox.com wrote: > On Fri, Feb 26, 2016 at 05:44:07AM -0700, crigariba wrote: >> We have a Supermicro chassis with a LSI 3008 HBA card. Installed FreeNas >> 9.3-RELEASE, but realized there were only 35 disks visible to the OS. The >> missing disk is in bay 24 (starting at 0 - 1st disk on rear backplane). >> >> Both in the card bios and booting a live gnu/linux shows all 36 disks and >> said disk is functional. > > Best practice for ZFS is to have different system and data pools. Your > system pool will have a small number of disks compared to your data pools, > and that tends to avoid issues like this one. There are other benefits > as well. Maybe in this case FreeNAS is just on flash, that seems the normal way to install it. You said sas2ircu saw no adapters, but I think you probably need to use sas3ircu for this model of HBA. Another quick test I suppose might be to boot with current NAS4Free - whether you want to use that or not, it's based on FreeBSD 10.2 so you might see some different behaviour. Though a FreeBSD 10.2 livecd/liveusb would achieve the same thing. Graham -- From owner-freebsd-scsi@freebsd.org Fri Feb 26 20:36:18 2016 Return-Path: Delivered-To: freebsd-scsi@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0B13AB4289 for ; Fri, 26 Feb 2016 20:36:18 +0000 (UTC) (envelope-from stephen.mcconnell@broadcom.com) Received: from mail-pa0-x229.google.com (mail-pa0-x229.google.com [IPv6:2607:f8b0:400e:c03::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA1B383B for ; Fri, 26 Feb 2016 20:36:18 +0000 (UTC) (envelope-from stephen.mcconnell@broadcom.com) Received: by mail-pa0-x229.google.com with SMTP id fl4so56371126pad.0 for ; Fri, 26 Feb 2016 12:36:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:references:in-reply-to:subject:date:message-id:mime-version :content-transfer-encoding:thread-index:content-language; bh=rQu6KiPOpMqB7vNWefnGT778bm+Wb8/TwTYt4yK50v4=; b=TtktcjWCPkQ9sNkUGr7BnM6bwBRQkLD40kVPP6jHKentf1JCjZki+TsZk7LsnfrODA zSLwTtmq7Bgfu8FaXZp+l26WK4k8s3wTdAqTGSb1ek7gSMVQRMg+rE6zDSItIwSxxvY7 jQZeadMXaLZrKXVwg2DTffOI3Pqy9rzmBfliU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:references:in-reply-to:subject:date :message-id:mime-version:content-transfer-encoding:thread-index :content-language; bh=rQu6KiPOpMqB7vNWefnGT778bm+Wb8/TwTYt4yK50v4=; b=g13hFdNvc2jDBFR6uLINK/HWHQl5s+AVun0reQ+th5/rWKtUmrT8jjG2AB2Ho9NNwd e3ZZK6vFfUcnML63ZzKf1I98phDLo2bqbwP/gT79Cc2+Z4yTNW4600K8MF0Td3j0t1P0 270Lqm/S9E5F5v5Pd9ooYx8E5Jpfqwsq0XeOe67T7675KJmgMLg9ZWmAXOyPsNc7gNJx xE/Lq3gUlGAJzHrxgC2mhQGDMWPajyXTSP0H27BGywvL+kqh0kGxKN/uZT2/jtzq7Q3Z 7RkRW4xMzwJ1sA18krLVxTczEUAqI0EfkYiNFc6wKi/pn/p0EKHyRv/VZMP661rk4fgC fU0g== X-Gm-Message-State: AD7BkJJ7U87++hJipu2kWfJGYAjkt3IkslAQzbaUrn98jTCMoYCzMgNkd9F98a6ailGpaS8d X-Received: by 10.66.132.73 with SMTP id os9mr4855559pab.27.1456518977813; Fri, 26 Feb 2016 12:36:17 -0800 (PST) Received: from C5SDN12 ([192.19.220.253]) by smtp.gmail.com with ESMTPSA id m5sm21248478pfi.84.2016.02.26.12.36.15 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 26 Feb 2016 12:36:16 -0800 (PST) From: "Stephen McConnell" To: "'crigariba'" , References: <54822835.3080800@crystal.harvard.edu> <548B4C25.8010501@crystal.harvard.edu> <1456490647564-6079717.post@n5.nabble.com> <20160226152657.GA75063@neutralgood.org> <56D0704D.1080906@physics.umn.edu> In-Reply-To: <56D0704D.1080906@physics.umn.edu> Subject: RE: LSI SAS 3008 card - 35 out of 36 disks detected Date: Fri, 26 Feb 2016 13:36:13 -0700 Message-ID: <053001d170d5$576a6330$063f2990$@broadcom.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQHzbsQAgHnUyrsml0jKHnNERhU5lwIU693fAifsZ98B31G6LwKohI4NAfWPcl0B2NpGGgMGlvKQAkZLaS4Byid/eZ5deXNQ Content-Language: en-us X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Feb 2016 20:36:19 -0000 > -----Original Message----- > From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- > scsi@freebsd.org] On Behalf Of Graham Allan > Sent: Friday, February 26, 2016 8:34 AM > To: freebsd-scsi@freebsd.org > Subject: Re: LSI SAS 3008 card - 35 out of 36 disks detected > > > > On 2/26/2016 9:26 AM, kpneal@pobox.com wrote: > > On Fri, Feb 26, 2016 at 05:44:07AM -0700, crigariba wrote: > >> We have a Supermicro chassis with a LSI 3008 HBA card. Installed > >> FreeNas 9.3-RELEASE, but realized there were only 35 disks visible to > >> the OS. The missing disk is in bay 24 (starting at 0 - 1st disk on rear > backplane). > >> > >> Both in the card bios and booting a live gnu/linux shows all 36 disks > >> and said disk is functional. This problem has to do with how the enclosure reports its slots to the HBA's Firmware. It sounds like you've researched it, so you probably know that. If it's possible for you to move your HBA to a Windows or Linux system, or if you can boot up in UEFI, I can give you access to a tool that will allow you to change the mapping mode on your HBA. There is a FreeBSD version, but it's not yet in a licensable form, and you will need to sign an EULA to use the tool. Let me know and I can send you a link and instructions. > > > > Best practice for ZFS is to have different system and data pools. Your > > system pool will have a small number of disks compared to your data > > pools, and that tends to avoid issues like this one. There are other > > benefits as well. > > Maybe in this case FreeNAS is just on flash, that seems the normal way to > install it. > > You said sas2ircu saw no adapters, but I think you probably need to use > sas3ircu for this model of HBA. > > Another quick test I suppose might be to boot with current NAS4Free - > whether you want to use that or not, it's based on FreeBSD 10.2 so you might > see some different behaviour. Though a FreeBSD 10.2 livecd/liveusb would > achieve the same thing. > > Graham > -- > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"