From owner-freebsd-hardware@freebsd.org Sun Jul 31 16:51:54 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B718BAAFE9; Sun, 31 Jul 2016 16:51:54 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0A6111772; Sun, 31 Jul 2016 16:51:53 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id u6VGpS1i058864; Sun, 31 Jul 2016 18:51:37 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id F41ADB18; Sun, 31 Jul 2016 18:51:27 +0200 (CEST) Message-ID: <579E2C8F.8090306@omnilan.de> Date: Sun, 31 Jul 2016 18:51:27 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Jason Zhang CC: freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, freebsd-hardware@freebsd.org Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i References: In-Reply-To: Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Sun, 31 Jul 2016 18:51:37 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Jul 2016 16:51:54 -0000 Bez=A8=B9glich Jason Zhang's Nachricht vom 17.06.2016 09:16 (localtime):= > Hi, > > I am working on storage service based on FreeBSD. I look forward to a = good result because many professional storage company use FreeBSD as its = OS. But I am disappointed with the Bad performance. I tested the the pe= rformance of LSI MegaRAID 9260-8i and had the following bad result: > > 1. Test environment: > (1) OS: FreeBSD 10.0 release > (2) Memory: 16G > (3) RAID adapter: LSI MegaRAID 9260-8i > (4) Disks: 9 SAS hard drives (10000 rpm), performance is expe= cted for each hard drive =20 Were the drives completely initialized? I remember that at least one vendor had implemented read-past-write for every sector when written first. It was with 15k 3.5" spindles and I'm really not sure which vendor it was, so I won't name any. But "slow init" had solved a similir problem for me back then... -Harry From owner-freebsd-hardware@freebsd.org Mon Aug 1 06:45:23 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 67342BACD35; Mon, 1 Aug 2016 06:45:23 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 281C81A8F; Mon, 1 Aug 2016 06:45:22 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.85) with esmtps (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (envelope-from ) id <1bU6yM-000mWX-2P>; Mon, 01 Aug 2016 08:45:10 +0200 Received: from p578a69f9.dip0.t-ipconnect.de ([87.138.105.249] helo=freyja.zeit4.iv.bundesimmobilien.de) by inpost2.zedat.fu-berlin.de (Exim 4.85) with esmtpsa (TLSv1.2:AES256-GCM-SHA384:256) (envelope-from ) id <1bU6yL-001051-PK>; Mon, 01 Aug 2016 08:45:09 +0200 Date: Mon, 1 Aug 2016 08:45:04 +0200 From: "O. Hartmann" To: Borja Marcos Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-hardware@freebsd.org, freebsd-stable@freebsd.org Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i Message-ID: <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> Organization: FU Berlin X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: 87.138.105.249 X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 06:45:23 -0000 On Wed, 22 Jun 2016 08:58:08 +0200 Borja Marcos wrote: > > On 22 Jun 2016, at 04:08, Jason Zhang wrote: > >=20 > > Mark, > >=20 > > Thanks > >=20 > > We have same RAID setting both on FreeBSD and CentOS including cache > > setting. In FreeBSD, I enabled the write cache but the performance is = the > > same. =20 > >=20 > > We don=E2=80=99t use ZFS or UFS, and test the performance on the RAW GE= OM disk > > =E2=80=9Cmfidx=E2=80=9D exported by mfi driver. We observed the =E2=80= =9Cgstat=E2=80=9D result and found > > that the write latency is too high. When we =E2=80=9Cdd" the disk with= 8k, it is > > lower than 1ms, but it is 6ms on 64kb write. It seems that each single > > write operation is very slow. But I don=E2=80=99t know whether it is a = driver > > problem or not. =20 >=20 > There is an option you can use (I do it all the time!) to make the card > behave as a plain HBA so that the disks are handled by the =E2=80=9Cda=E2= =80=9D driver.=20 >=20 > Add this to /boot/loader.conf >=20 > hw.mfi.allow_cam_disk_passthrough=3D1 > mfip_load=3D=E2=80=9CYES" >=20 > And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To avoid co= nfusions, it=E2=80=99s > better to make sure the disks are not part of a =E2=80=9Cjbod=E2=80=9D or= logical volume > configuration. >=20 >=20 >=20 >=20 > Borja. [...] How is this supposed to work when ALL disks (including boot device) are set= tled with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 and detected within FreeBSD 11-BETA and 12-CURRENT) controller itself? I did not find any solution to force the CP400i into a mode making itself acting as a HBA (we intend to use all drives with ZFS and let FreeBSD kernel/ZFS control everything). The boot device is a 256 GB Samsung SSD for enterprise use and putting the = UEFI load onto a EFI partition from 11-CURRENT-ALPHA4 is worse: dd takes up to almost a minute to put the image onto the SSD. The SSD active LED is blinki= ng alle the time indicating activity. Caches are off. I tried to enable the ca= che via the mfiutil command by 'mfiutil cache mfid0 enable', but it failed ... = It failed also on all other attached drives. I didn't further go into more investigations right now, since the experience with the EFI boot loader makes me suspect bad performance and that is harsh= so to speak. Glad to have found this thread anyway. I cross post this also to CURRENT as it might be an issue with CURRENT ... Kind regards, Oliver Hartmann From owner-freebsd-hardware@freebsd.org Mon Aug 1 09:56:44 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD56EBAA2C5; Mon, 1 Aug 2016 09:56:44 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3AE8E1812; Mon, 1 Aug 2016 09:56:43 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id 5B5319DC575; Mon, 1 Aug 2016 11:48:31 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Borja Marcos In-Reply-To: <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> Date: Mon, 1 Aug 2016 11:48:30 +0200 Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> To: "O. Hartmann" X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 09:56:44 -0000 > On 01 Aug 2016, at 08:45, O. Hartmann = wrote: >=20 > On Wed, 22 Jun 2016 08:58:08 +0200 > Borja Marcos wrote: >=20 >> There is an option you can use (I do it all the time!) to make the = card >> behave as a plain HBA so that the disks are handled by the =E2=80=9Cda=E2= =80=9D driver.=20 >>=20 >> Add this to /boot/loader.conf >>=20 >> hw.mfi.allow_cam_disk_passthrough=3D1 >> mfip_load=3D=E2=80=9CYES" >>=20 >> And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To = avoid confusions, it=E2=80=99s >> better to make sure the disks are not part of a =E2=80=9Cjbod=E2=80=9D = or logical volume >> configuration. >>=20 >>=20 >>=20 >>=20 >> Borja. > [...] >=20 > How is this supposed to work when ALL disks (including boot device) = are settled > with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 = and > detected within FreeBSD 11-BETA and 12-CURRENT) controller itself? >=20 > I did not find any solution to force the CP400i into a mode making = itself > acting as a HBA (we intend to use all drives with ZFS and let FreeBSD > kernel/ZFS control everything). Have you tried that particular option?=20 With kinda recent LSI based cards you have three options: - The most usual and definitely NOT RECOMMENDED option is to define a = logical volume per disk which actually LSI Logic called before JBOD mode. It=E2=80=99s not = recommended at all if you want to run ZFS. - Recent cards, I think I saw this first on the LSI3008, have a JBOD = mode that exposes the drives as =E2=80=9Cmfisyspd=E2=80=9D devices. I don=E2=80=99t recommend it either, because the syspd drives are a sort = of limited version of a disk device. With SSDs, especially, you don=E2=80=99t have access to the TRIM command. - The third option is to make the driver expose the SAS devices like a = HBA would do, so that they are visible to the CAM layer, and disks are handled by the stock =E2=80=9Cda=E2=80=9D = driver, which is the ideal solution.=20 However, this third option might not be available in some custom = firmware versions for certain manufacturers? I don=C2=B4t know. And I would hesitate to make the conversion on a production = machine unless you have a complete and reliable full backup of all the data in case you need to rebuild it. In order to do it you need a couple of things. You need to set the = variable hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. When booting installation media, enter command mode and use these = commands: ----- set hw.mfi.allow_cam_disk_passthrough=3D1 load mfip boot =E2=80=94=E2=80=94=E2=80=94 Remember that after installation you need to update /boot/loader.conf in = the system you just installed with the following contents: hw.mfi.allow_cam_disk_passthrough=3D1 mfip_load=3D=E2=80=9CYES=E2=80=9D A note regarding CAM and MFI visibility: On some old firmware versions = for the LSI2008 I=E2=80=99ve even seen the disks available both as =E2=80=9Cmfi=E2=80=9D and =E2=80=9Cda=E2=80=9D = drivers. If possible, you should try to set them up as =E2=80=9Cunconfigur= ed good=E2=80=9D on the RAID firmware. Use the RAID firmware set up or maybe mfiutil(8) Also, make sure you don=E2=80=99t create any logical volumes on the = disks you want exposed to CAM. You should delete the logical volumes so that the MFI firmware doesn=E2=80=99t do anything = with them.=20 AND BEWARE: Doing these changes to a system in production with valuable = data is dangerous. Make sure you have a full and sound backup before making these changes. As a worst case, the card could expose the devices both as =E2=80=9Csyspd=E2= =80=9D and CAM (i.e., =E2=80=9Cda=E2=80=9D drives) but as long as you = don=E2=80=99t touch the syspd devices the card won=E2=80=99t do anything to them as = far as I know. It could be a serious problem, however, if you=20 access a drive part of a logical volume through CAM, as RAID cards tend = do to =E2=80=9Cpatrol reads=E2=80=9D and other stuff on them.=20 Provided it=E2=80=99s safe to do what I recommended, try it and follow = up by email.=20 Borja. From owner-freebsd-hardware@freebsd.org Mon Aug 1 13:12:09 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31D76BA7AFD; Mon, 1 Aug 2016 13:12:09 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E526F15E9; Mon, 1 Aug 2016 13:12:08 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost.zedat.fu-berlin.de (Exim 4.85) with esmtps (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (envelope-from ) id <1bUD0m-0037tt-6I>; Mon, 01 Aug 2016 15:12:04 +0200 Received: from p578a69f9.dip0.t-ipconnect.de ([87.138.105.249] helo=freyja.zeit4.iv.bundesimmobilien.de) by inpost2.zedat.fu-berlin.de (Exim 4.85) with esmtpsa (TLSv1.2:AES256-GCM-SHA384:256) (envelope-from ) id <1bUD0l-001Y2q-Rz>; Mon, 01 Aug 2016 15:12:04 +0200 Date: Mon, 1 Aug 2016 15:12:03 +0200 From: "O. Hartmann" To: Borja Marcos Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i Message-ID: <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> Organization: FU Berlin X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: 87.138.105.249 X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 13:12:09 -0000 On Mon, 1 Aug 2016 11:48:30 +0200 Borja Marcos wrote: Hello. First, thanks for responding so quickly. > > On 01 Aug 2016, at 08:45, O. Hartmann wro= te: > >=20 > > On Wed, 22 Jun 2016 08:58:08 +0200 > > Borja Marcos wrote: > > =20 > >> There is an option you can use (I do it all the time!) to make the card > >> behave as a plain HBA so that the disks are handled by the =E2=80=9Cda= =E2=80=9D driver.=20 > >>=20 > >> Add this to /boot/loader.conf > >>=20 > >> hw.mfi.allow_cam_disk_passthrough=3D1 > >> mfip_load=3D=E2=80=9CYES" > >>=20 > >> And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To avoid= confusions, it=E2=80=99s > >> better to make sure the disks are not part of a =E2=80=9Cjbod=E2=80=9D= or logical volume > >> configuration. > >>=20 > >>=20 > >>=20 > >>=20 > >> Borja. =20 > > [...] > >=20 > > How is this supposed to work when ALL disks (including boot device) are > > settled with the mfi (in our case, it is a Fujitsu CP400i, based upon > > LSI3008 and detected within FreeBSD 11-BETA and 12-CURRENT) controller > > itself? > >=20 > > I did not find any solution to force the CP400i into a mode making itse= lf > > acting as a HBA (we intend to use all drives with ZFS and let FreeBSD > > kernel/ZFS control everything). =20 >=20 > Have you tried that particular option?=20 I have, indeed, used the "JBOD" function of the PRAID CP400i controller and= the intention of my posting regards to the suspicion, that this is, as mentione= d in many posts concerning RAID controllers and ZFS, the reason for the worse performance. And as I can see, it has been confirmed, sadly. >=20 > With kinda recent LSI based cards you have three options: >=20 > - The most usual and definitely NOT RECOMMENDED option is to define a log= ical > volume per disk which actually LSI Logic called before JBOD mode. It=E2= =80=99s not > recommended at all if you want to run ZFS. This is the only way to expose each disk as it is to the OS with the PRAID CP400i built-in into our RX1330-M2 server (XEON Skylake based). I ordered t= hat specific box with a HBA capable controller. Searching the net reveals that there is another one, called PSAS CP400i, which is also based on LSI/Avago SAS3008 and the possibility to expose drives as-is is explicitely mentioned= . I do not know whether this is a software feature - as I suspect - or something which has been hardwired to the controller. >=20 > - Recent cards, I think I saw this first on the LSI3008, have a JBOD mode > that exposes the drives as =E2=80=9Cmfisyspd=E2=80=9D devices. I don=E2= =80=99t recommend it either, > because the syspd drives are a sort of limited version of a disk device. = With > SSDs, especially, you don=E2=80=99t have access to the TRIM command. They expose the drives as "mfidX" if setup as JBOD. >=20 > - The third option is to make the driver expose the SAS devices like a HBA > would do, so that they are visible to the CAM layer, and disks are handle= d by > the stock =E2=80=9Cda=E2=80=9D driver, which is the ideal solution.=20 I didn't find any switch which offers me the opportunity to put the PRAID CP400i into a simple HBA mode. =20 >=20 > However, this third option might not be available in some custom firmware > versions for certain manufacturers? I don=C2=B4t know. And I would hesita= te to > make the conversion on a production machine unless you have a complete and > reliable full backup of all the data in case you need to rebuild it. The boxes are empty and ready-for-installation, so I do not worry. It is mo= re worrying about this stupid software-based strangulations of options by Fuji= tsu - if any. i do not want to blame them before I haven't double-checked. >=20 > In order to do it you need a couple of things. You need to set the variab= le > hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. >=20 > When booting installation media, enter command mode and use these command= s: >=20 > ----- > set hw.mfi.allow_cam_disk_passthrough=3D1 > load mfip > boot > =E2=80=94=E2=80=94=E2=80=94 Well, I'm truly aware of this problemacy and solution (now), but I run into= a henn-egg-problem, literally. As long as I can boot off of the installation medium, I have a kernel which deals with the setting. But the boot medium is supposed to be a SSD sitting with the PRAID CP400i controller itself! So, I never be able to boot off the system without crippling the ability to have a fullspeed ZFS configuration which I suppose to have with HBA mode, but not with any of the forced RAID modes offered by the controller.=20 I will check with Fujitsu for a solution. Maybe the PRAID CP400i is capable somehow of being a PSAS CP400i also, even if not exposed by the recent/installed firmware. Kind regards, Oliver=20 =20 >=20 >=20 > Remember that after installation you need to update /boot/loader.conf in = the > system you just installed with the following contents: >=20 > hw.mfi.allow_cam_disk_passthrough=3D1 > mfip_load=3D=E2=80=9CYES=E2=80=9D >=20 >=20 > A note regarding CAM and MFI visibility: On some old firmware versions for > the LSI2008 I=E2=80=99ve even seen the disks available both as =E2=80=9Cm= fi=E2=80=9D and =E2=80=9Cda=E2=80=9D > drivers. If possible, you should try to set them up as =E2=80=9Cunconfigu= red good=E2=80=9D on > the RAID firmware. Use the RAID firmware set up or maybe mfiutil(8) >=20 > Also, make sure you don=E2=80=99t create any logical volumes on the disks= you want > exposed to CAM. You should delete the logical volumes so that the MFI > firmware doesn=E2=80=99t do anything with them.=20 >=20 > AND BEWARE: Doing these changes to a system in production with valuable d= ata > is dangerous. Make sure you have a full and sound backup before making th= ese > changes. >=20 > As a worst case, the card could expose the devices both as =E2=80=9Csyspd= =E2=80=9D and CAM > (i.e., =E2=80=9Cda=E2=80=9D drives) but as long as you don=E2=80=99t touc= h the syspd devices the card > won=E2=80=99t do anything to them as far as I know. It could be a serious= problem, > however, if you access a drive part of a logical volume through CAM, as R= AID > cards tend do to =E2=80=9Cpatrol reads=E2=80=9D and other stuff on them.= =20 >=20 > Provided it=E2=80=99s safe to do what I recommended, try it and follow up= by email.=20 >=20 >=20 >=20 >=20 >=20 > Borja. >=20 >=20 >=20 >=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= .org" From owner-freebsd-hardware@freebsd.org Mon Aug 1 13:30:09 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 816A2BA7F48; Mon, 1 Aug 2016 13:30:09 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 06F271D5B; Mon, 1 Aug 2016 13:30:08 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id D14279DCCC2; Mon, 1 Aug 2016 15:29:58 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Borja Marcos In-Reply-To: <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> Date: Mon, 1 Aug 2016 15:29:58 +0200 Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> To: "O. Hartmann" X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 13:30:09 -0000 > On 01 Aug 2016, at 15:12, O. Hartmann = wrote: >=20 > First, thanks for responding so quickly. >=20 >> - The third option is to make the driver expose the SAS devices like = a HBA >> would do, so that they are visible to the CAM layer, and disks are = handled by >> the stock =E2=80=9Cda=E2=80=9D driver, which is the ideal solution.=20= >=20 > I didn't find any switch which offers me the opportunity to put the = PRAID > CP400i into a simple HBA mode. The switch is in the FreeBSD mfi driver, the loader tunable I mentioned, = regardless of what the card firmware does or pretends to do. It=E2=80=99s not visible doing a "sysctl -a=E2=80=9D, but it exists and = it=E2=80=99s unique even. It=E2=80=99s defined here: = https://svnweb.freebsd.org/base/stable/10/sys/dev/mfi/mfi_cam.c?revision=3D= 267084&view=3Dmarkup (line 93) >> In order to do it you need a couple of things. You need to set the = variable >> hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. >>=20 >> When booting installation media, enter command mode and use these = commands: >>=20 >> ----- >> set hw.mfi.allow_cam_disk_passthrough=3D1 >> load mfip >> boot >> =E2=80=94=E2=80=94=E2=80=94 >=20 > Well, I'm truly aware of this problemacy and solution (now), but I run = into a > henn-egg-problem, literally. As long as I can boot off of the = installation > medium, I have a kernel which deals with the setting. But the boot = medium is > supposed to be a SSD sitting with the PRAID CP400i controller itself! = So, I > never be able to boot off the system without crippling the ability to = have a > fullspeed ZFS configuration which I suppose to have with HBA mode, but = not > with any of the forced RAID modes offered by the controller.=20 Been there plenty of times, even argued quite strongly about the = advantages of ZFS against hardware based RAID 5 cards. :) I remember when the Dell salesmen couldn=E2=80=99t possibly = understand why I wanted a =E2=80=9Csoftware based RAID rather than a robust, hardware based solution=E2=80=9D :D=20 At worst, you can set up a simple boot from a thumb drive or, even = better, a SATADOM installed inside the server. I guess it will have SATA ports on the mainboard. That=E2=80=99s what I use to do. = FreeNAS uses a similar approach as well. And some modern servers also can boot from a SD card which you can use just to load the kernel. Depending on the number of disks you have, you can also sacrifice two to = set up a mirror with a =E2=80=9Cnomal=E2=80=9D boot system, and using the rest of the disks for ZFS. Actually I=E2=80=99ve got an old server I = set up in 2012. It has 16 disks, and I created a logical volume (mirror) with 2 disks for boot, the other 14 disks for ZFS. If I installed this server now I would do it different, booting off a = thumb drive. But I was younger and naiver :) Borja. From owner-freebsd-hardware@freebsd.org Mon Aug 1 20:34:06 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15F34BAB1F3; Mon, 1 Aug 2016 20:34:06 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (mail.sorbs.net [67.231.146.200]) by mx1.freebsd.org (Postfix) with ESMTP id 03ADB1358; Mon, 1 Aug 2016 20:34:05 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 8BIT Content-type: text/plain; charset=UTF-8; format=flowed Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0OB800C6YRNTJH00@hades.sorbs.net>; Mon, 01 Aug 2016 10:38:21 -0700 (PDT) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i To: Borja Marcos , "O. Hartmann" Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> From: Michelle Sullivan Message-id: <579F8743.8030104@sorbs.net> Date: Mon, 01 Aug 2016 19:30:43 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40 In-reply-to: <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 20:34:06 -0000 Borja Marcos wrote: >> On 01 Aug 2016, at 15:12, O. Hartmann wrote: >> >> First, thanks for responding so quickly. >> >>> - The third option is to make the driver expose the SAS devices like a HBA >>> would do, so that they are visible to the CAM layer, and disks are handled by >>> the stock “da” driver, which is the ideal solution. >> I didn't find any switch which offers me the opportunity to put the PRAID >> CP400i into a simple HBA mode. > The switch is in the FreeBSD mfi driver, the loader tunable I mentioned, regardless of what the card > firmware does or pretends to do. > > It’s not visible doing a "sysctl -a”, but it exists and it’s unique even. It’s defined here: > > https://svnweb.freebsd.org/base/stable/10/sys/dev/mfi/mfi_cam.c?revision=267084&view=markup > (line 93) > >>> In order to do it you need a couple of things. You need to set the variable >>> hw.mfi.allow_cam_disk_passthrough=1 and to load the mfip.ko module. >>> >>> When booting installation media, enter command mode and use these commands: >>> >>> ----- >>> set hw.mfi.allow_cam_disk_passthrough=1 >>> load mfip >>> boot >>> ——— >> Well, I'm truly aware of this problemacy and solution (now), but I run into a >> henn-egg-problem, literally. As long as I can boot off of the installation >> medium, I have a kernel which deals with the setting. But the boot medium is >> supposed to be a SSD sitting with the PRAID CP400i controller itself! So, I >> never be able to boot off the system without crippling the ability to have a >> fullspeed ZFS configuration which I suppose to have with HBA mode, but not >> with any of the forced RAID modes offered by the controller. > Been there plenty of times, even argued quite strongly about the advantages of ZFS against hardware based RAID > 5 cards. :) I remember when the Dell salesmen couldn’t possibly understand why I wanted a “software based RAID rather than a > robust, hardware based solution” :D There are reasons for using either... Nowadays its seems the conversations have degenerated into those like Windows vs Linux vs Mac where everyone thinks their answer is the right one (just as you suggested you (Borja Marcos) did with the Dell salesman), where in reality each has its own advantages and disadvantages. Eg: I'm running 2 zfs servers on 'LSI 9260-16i's... big mistake! (the ZFS, not LSI's)... one is a 'movie server' the other a 'postgresql database' server... The latter most would agree is a bad use of zfs, the die-hards won't but then they don't understand database servers and how they work on disk. The former has mixed views, some argue that zfs is the only way to ensure the movies will always work, personally I think of all the years before zfs when my data on disk worked without failure until the disks themselves failed... and RAID stopped that happening... what suddenly changed, are disks and ram suddenly not reliable at transferring data? .. anyhow back to the issue there is another part with this particular hardware that people just throw away... The LSI 9260-* controllers have been designed to provide on hardware RAID. The caching whether using the Cachecade SSD or just oneboard ECC memory is *ONLY* used when running some sort of RAID set and LVs... this is why LSI recommend 'MegaCli -CfgEachDskRaid0' because it does enable caching.. A good read on how to setup something similar is here: https://calomel.org/megacli_lsi_commands.html (disclaimer, I haven't parsed it all so the author could be clueless, but it seems to give generally good advice.) Going the way of 'JBOD' is a bad thing to do, just don't, performance sucks. As for the recommended command above, can't comment because currently I don't use it nor will I need to in the near future... but... If you (O Hartmann) want to use or need to use ZFS with any OS including FreeBSD don't go with the LSI 92xx series controllers, its just the wrong thing to do.. Pick an HBA that is designed to give you direct access to the drives not one you have to kludge and cajole.. Including LSI controllers with caches that use the mfi driver, just not those that are not designed to work in a non RAID mode (with or without the passthru command/mode above.) > > At worst, you can set up a simple boot from a thumb drive or, even better, a SATADOM installed inside the server. I guess it will > have SATA ports on the mainboard. That’s what I use to do. FreeNAS uses a similar approach as well. And some modern servers > also can boot from a SD card which you can use just to load the kernel. > > Depending on the number of disks you have, you can also sacrifice two to set up a mirror with a “nomal” boot system, and using > the rest of the disks for ZFS. Actually I’ve got an old server I set up in 2012. It has 16 disks, and I created a logical volume (mirror) > with 2 disks for boot, the other 14 disks for ZFS. > > If I installed this server now I would do it different, booting off a thumb drive. But I was younger and naiver :) > > If I installed mine now I would do them differently as well... neither would run ZFS, both would use their on card RAID kernels and UFS on top of them... ZFS would be reserved for the multi-user NFS file servers. (and trust me here, when it comes to media servers - where the media is just stored not changed/updated/edited - the 16i with a good highspeed SSD as 'Cachecade' really performs well... and on a moderately powerful MB/CPU combo with good RAM and several gigabit interfaces it's surprising how many unicast transcoded media streams it can handle... (read: my twin fibres are saturated before the machine reaches anywhere near full load, and I can still write at 13MBps from my old Mac Mini over NFS... which is about all it can do without any load either.) So moral of the story/choices. Don't go with ZFS because people tell you its best, because it isn't, go with ZFS if it suits your hardware and application, and if ZFS suits your application, get hardware for it. Regards, -- Michelle Sullivan http://www.mhix.org/ From owner-freebsd-hardware@freebsd.org Tue Aug 2 03:22:39 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C906BAC19A; Tue, 2 Aug 2016 03:22:39 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: from mail-yw0-x22e.google.com (mail-yw0-x22e.google.com [IPv6:2607:f8b0:4002:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1487F19F7; Tue, 2 Aug 2016 03:22:39 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: by mail-yw0-x22e.google.com with SMTP id r9so190568492ywg.0; Mon, 01 Aug 2016 20:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=EhBNUcs8lgqhpBnZ5R5QKsQsfkBSkZLt4s92Cr+/9VQ=; b=h8/1FO4JPGOd+vLjNn4QO7CdoT5S2QiLxwnh7WrLzBEAa7dJYWsu9zpb//sbon9XKw n4yRbYKeOkbV+aYMqNXBZKdob8L7VYEiXpA0ZmDOhfkYq/aaDQB406i89nyex+AO1euN cLXtzW85+S9izeALjMWBayd09JRlg+kA0AgQvzZhtXsSzWBkneNvbjoix3CDlC4jabtr +2cHPZZkSSXLKyLS+tXXekyfteYcgBC4NPGRaEy+mcAlBiNukgEMs0x9tmODmuK8ukyr 86uFcRJU5Ven8S3zygEgIifI9eMAQO1SmosrcHm8mPaaec2+6eEb7oZgIvZU0pwZ+d7Y 1+jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=EhBNUcs8lgqhpBnZ5R5QKsQsfkBSkZLt4s92Cr+/9VQ=; b=OW6HfflMb8+jvPf9JZTOHhMf25nkyHdgqiPxEVELPz3HUxbVXXJD9qs9khC6gbZ7DD /h+Vc7aj+Z2xENhKZUZhfbcRUnqAieLUWnzQVdgo8PdBxNHD00Wav9GpZJTL8wzw11Wt gB+icUDGGTDIc5UZFqW9gsa8gs6EVxPYTaZ9srJinwC/etl0V/9tx3flDdQV1EzJULH9 xyppIgQVe29ZlqCIYL912f/34d4VmAV41x4YZMGU7O5oI8sSoDgTy8wnD6v3BpbTHNBq xlkIXiOTgCT9FL/3XaKHmzdk3YgYql8w/TnCI73e62zc6np7k6dyRwQ1WzmMfo8r+7Fw G6ZA== X-Gm-Message-State: AEkoouulBy9374uIPj/TnlS4RZ9vZ+R/mPKnGIzZpAOA4eD7nZt4XUXBrh62cy8BY8Nt5dXcE+COzQ3QOMfQpw== X-Received: by 10.37.201.131 with SMTP id z125mr43845470ybf.183.1470108158170; Mon, 01 Aug 2016 20:22:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.51.150 with HTTP; Mon, 1 Aug 2016 20:22:37 -0700 (PDT) In-Reply-To: <579F8743.8030104@sorbs.net> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> <579F8743.8030104@sorbs.net> From: Ultima Date: Mon, 1 Aug 2016 23:22:37 -0400 Message-ID: Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i To: Michelle Sullivan Cc: Borja Marcos , "O. Hartmann" , Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 03:22:39 -0000 If anyone is interested, as Michelle Sullivan just mentioned. One problem I found when looking for an HBA is that they are not so easy to find. Scoured the internet for a backup HBA I came across these - http://www.avagotech.com/products/server-storage/host-bus-adapters/#tab-12G= b1 Can only speak for sas-9305-24i. All 24 bays are occupied and quite pleased with the performance compared to its predecessor. It was originally going to be a backup unit, however that changed after running a scrub and the amount of hours to complete cut in half (around 30ish to 15 for 35T). And of course, the reason for this post, it replaced a raid card in passthrough mode. Another note, because it is an HBA, the ability to flash firmware is once again possible! (yay!) +1 to HBA's + ZFS, if possible replace it for an HBA. On Mon, Aug 1, 2016 at 1:30 PM, Michelle Sullivan wrote: > Borja Marcos wrote: > >> On 01 Aug 2016, at 15:12, O. Hartmann >>> wrote: >>> >>> First, thanks for responding so quickly. >>> >>> - The third option is to make the driver expose the SAS devices like a >>>> HBA >>>> would do, so that they are visible to the CAM layer, and disks are >>>> handled by >>>> the stock =E2=80=9Cda=E2=80=9D driver, which is the ideal solution. >>>> >>> I didn't find any switch which offers me the opportunity to put the PRA= ID >>> CP400i into a simple HBA mode. >>> >> The switch is in the FreeBSD mfi driver, the loader tunable I mentioned, >> regardless of what the card >> firmware does or pretends to do. >> >> It=E2=80=99s not visible doing a "sysctl -a=E2=80=9D, but it exists and = it=E2=80=99s unique even. >> It=E2=80=99s defined here: >> >> >> https://svnweb.freebsd.org/base/stable/10/sys/dev/mfi/mfi_cam.c?revision= =3D267084&view=3Dmarkup >> (line 93) >> >> In order to do it you need a couple of things. You need to set the >>>> variable >>>> hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. >>>> >>>> When booting installation media, enter command mode and use these >>>> commands: >>>> >>>> ----- >>>> set hw.mfi.allow_cam_disk_passthrough=3D1 >>>> load mfip >>>> boot >>>> =E2=80=94=E2=80=94=E2=80=94 >>>> >>> Well, I'm truly aware of this problemacy and solution (now), but I run >>> into a >>> henn-egg-problem, literally. As long as I can boot off of the >>> installation >>> medium, I have a kernel which deals with the setting. But the boot >>> medium is >>> supposed to be a SSD sitting with the PRAID CP400i controller itself! >>> So, I >>> never be able to boot off the system without crippling the ability to >>> have a >>> fullspeed ZFS configuration which I suppose to have with HBA mode, but >>> not >>> with any of the forced RAID modes offered by the controller. >>> >> Been there plenty of times, even argued quite strongly about the >> advantages of ZFS against hardware based RAID >> 5 cards. :) I remember when the Dell salesmen couldn=E2=80=99t possibly >> understand why I wanted a =E2=80=9Csoftware based RAID rather than a >> robust, hardware based solution=E2=80=9D :D >> > > There are reasons for using either... > > Nowadays its seems the conversations have degenerated into those like > Windows vs Linux vs Mac where everyone thinks their answer is the right o= ne > (just as you suggested you (Borja Marcos) did with the Dell salesman), > where in reality each has its own advantages and disadvantages. Eg: I'm > running 2 zfs servers on 'LSI 9260-16i's... big mistake! (the ZFS, not > LSI's)... one is a 'movie server' the other a 'postgresql database' > server... The latter most would agree is a bad use of zfs, the die-hards > won't but then they don't understand database servers and how they work o= n > disk. The former has mixed views, some argue that zfs is the only way to > ensure the movies will always work, personally I think of all the years > before zfs when my data on disk worked without failure until the disks > themselves failed... and RAID stopped that happening... what suddenly > changed, are disks and ram suddenly not reliable at transferring data? .. > anyhow back to the issue there is another part with this particular > hardware that people just throw away... > > The LSI 9260-* controllers have been designed to provide on hardware > RAID. The caching whether using the Cachecade SSD or just oneboard ECC > memory is *ONLY* used when running some sort of RAID set and LVs... this = is > why LSI recommend 'MegaCli -CfgEachDskRaid0' because it does enable > caching.. A good read on how to setup something similar is here: > https://calomel.org/megacli_lsi_commands.html (disclaimer, I haven't > parsed it all so the author could be clueless, but it seems to give > generally good advice.) Going the way of 'JBOD' is a bad thing to do, ju= st > don't, performance sucks. As for the recommended command above, can't > comment because currently I don't use it nor will I need to in the near > future... but... > > If you (O Hartmann) want to use or need to use ZFS with any OS including > FreeBSD don't go with the LSI 92xx series controllers, its just the wrong > thing to do.. Pick an HBA that is designed to give you direct access to > the drives not one you have to kludge and cajole.. Including LSI > controllers with caches that use the mfi driver, just not those that are > not designed to work in a non RAID mode (with or without the passthru > command/mode above.) > > > > >> At worst, you can set up a simple boot from a thumb drive or, even >> better, a SATADOM installed inside the server. I guess it will >> have SATA ports on the mainboard. That=E2=80=99s what I use to do. FreeN= AS uses a >> similar approach as well. And some modern servers >> also can boot from a SD card which you can use just to load the kernel. >> >> Depending on the number of disks you have, you can also sacrifice two to >> set up a mirror with a =E2=80=9Cnomal=E2=80=9D boot system, and using >> the rest of the disks for ZFS. Actually I=E2=80=99ve got an old server I= set up >> in 2012. It has 16 disks, and I created a logical volume (mirror) >> with 2 disks for boot, the other 14 disks for ZFS. >> >> If I installed this server now I would do it different, booting off a >> thumb drive. But I was younger and naiver :) >> >> >> > If I installed mine now I would do them differently as well... neither > would run ZFS, both would use their on card RAID kernels and UFS on top o= f > them... ZFS would be reserved for the multi-user NFS file servers. (and > trust me here, when it comes to media servers - where the media is just > stored not changed/updated/edited - the 16i with a good highspeed SSD as > 'Cachecade' really performs well... and on a moderately powerful MB/CPU > combo with good RAM and several gigabit interfaces it's surprising how ma= ny > unicast transcoded media streams it can handle... (read: my twin fibres a= re > saturated before the machine reaches anywhere near full load, and I can > still write at 13MBps from my old Mac Mini over NFS... which is about all > it can do without any load either.) > > So moral of the story/choices. Don't go with ZFS because people tell you > its best, because it isn't, go with ZFS if it suits your hardware and > application, and if ZFS suits your application, get hardware for it. > > Regards, > > -- > Michelle Sullivan > http://www.mhix.org/ > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-hardware@freebsd.org Tue Aug 2 08:27:08 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE388BAC21B; Tue, 2 Aug 2016 08:27:08 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 64B0518E3; Tue, 2 Aug 2016 08:27:07 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id B555C9DD613; Tue, 2 Aug 2016 10:27:03 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Borja Marcos In-Reply-To: <579F8743.8030104@sorbs.net> Date: Tue, 2 Aug 2016 10:27:02 +0200 Cc: "O. Hartmann" , Jason Zhang , freebsd-performance@freebsd.org, freebsd-current , FreeBSD-STABLE Mailing List , freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> <20160801151203.14a7a67d@freyja.zeit4.iv.bundesimmobilien.de> <0CA1A1F1-AFDD-4763-84C3-2FC059F44789@sarenet.es> <579F8743.8030104@sorbs.net> To: Michelle Sullivan X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 08:27:08 -0000 > On 01 Aug 2016, at 19:30, Michelle Sullivan = wrote: >=20 > There are reasons for using either=E2=80=A6 Indeed, but my decision was to run ZFS. And getting a HBA in some = configurations can be difficult because vendors insist on using=20 RAID adapters. After all, that=E2=80=99s what most of their customers = demand. Fortunately, at least some Avago/LSI cards can work as HBAs pretty well. = An example is the now venerable LSI2008. > Nowadays its seems the conversations have degenerated into those like = Windows vs Linux vs Mac where everyone thinks their answer is the right = one (just as you suggested you (Borja Marcos) did with the Dell = salesman), where in reality each has its own advantages and = disadvantages. I know, but this is not the case. But it=E2=80=99s quite frustrating to = try to order a server with a HBA rather than a RAID and receiving an = answer such as =E2=80=9Cthe HBA option is not available=E2=80=9D. That=E2=80=99s why = people are zapping, flashing and, generally, torturing HBA cards rather = cruelly ;) So, in my case, it=E2=80=99s not about what=E2=80=99s better or worse. = It=E2=80=99s just a simpler issue. Customer (myself) has made a = decision, which can be right or wrong. Manufacturer fails to deliver = what I need. If it was only one manufacturer, well, off with them, but = the issue is widespread in industry.=20 > Eg: I'm running 2 zfs servers on 'LSI 9260-16i's... big mistake! (the = ZFS, not LSI's)... one is a 'movie server' the other a 'postgresql = database' server... The latter most would agree is a bad use of zfs, = the die-hards won't but then they don't understand database servers and = how they work on disk. The former has mixed views, some argue that zfs = is the only way to ensure the movies will always work, personally I = think of all the years before zfs when my data on disk worked without = failure until the disks themselves failed... and RAID stopped that = happening... what suddenly changed, are disks and ram suddenly not = reliable at transferring data? .. anyhow back to the issue there is = another part with this particular hardware that people just throw = away=E2=80=A6 Well, silent corruption can happen. I=E2=80=99ve seen it once caused by = a flaky HBA and ZFS saved the cake. Yes. there were reliable replicas. = Still, rebuilding would be a pain in the ass.=20 > The LSI 9260-* controllers have been designed to provide on hardware = RAID. The caching whether using the Cachecade SSD or just oneboard ECC = memory is *ONLY* used when running some sort of RAID set and LVs... this = is why LSI recommend 'MegaCli -CfgEachDskRaid0' because it does enable = caching.. A good read on how to setup something similar is here: = https://calomel.org/megacli_lsi_commands.html (disclaimer, I haven't = parsed it all so the author could be clueless, but it seems to give = generally good advice.) Going the way of 'JBOD' is a bad thing to do, = just don't, performance sucks. As for the recommended command above, = can't comment because currently I don't use it nor will I need to in the = near future... but=E2=80=A6 Actually it=E2=80=99s not a good idea to use heavy disk caching when = running ZFS. Its reliability depends on being able to commit metadata to = disk. So I don=E2=80=99t care about that caching option. Provided you = have enough RAM, ZFS is very effective caching data itself. > If you (O Hartmann) want to use or need to use ZFS with any OS = including FreeBSD don't go with the LSI 92xx series controllers, its = just the wrong thing to do.. Pick an HBA that is designed to give you = direct access to the drives not one you have to kludge and cajole.. = Including LSI controllers with caches that use the mfi driver, just not = those that are not designed to work in a non RAID mode (with or without = the passthru command/mode above.) As I said, the problem is, sometimes it=E2=80=99s not so easy to find = the right HBA.=20 > So moral of the story/choices. Don't go with ZFS because people tell = you its best, because it isn't, go with ZFS if it suits your hardware = and application, and if ZFS suits your application, get hardware for it. Indeed, I second this. But really, "hardware for it" covers a rather = broad cathegory ;) ZFS can even manage to work on hardware _against_ it. Borja. From owner-freebsd-hardware@freebsd.org Fri Aug 5 10:32:50 2016 Return-Path: Delivered-To: freebsd-hardware@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 817F2BAF9EC for ; Fri, 5 Aug 2016 10:32:50 +0000 (UTC) (envelope-from Ben.Lavery@capita.co.uk) Received: from cluster-a.mailcontrol.com (cluster-a.mailcontrol.com [85.115.52.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.mailcontrol.com", Issuer "DigiCert SHA2 High Assurance Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E843D1A38 for ; Fri, 5 Aug 2016 10:32:48 +0000 (UTC) (envelope-from Ben.Lavery@capita.co.uk) Received: from CAPPRWMEEG02.capita.co.uk ([195.89.201.254]) by rly29a.srv.mailcontrol.com (MailControl) with ESMTPS id u759NmOR084022 (version=TLSv1 cipher=AES128-SHA bits=128 verify=OK) for ; Fri, 5 Aug 2016 10:24:29 +0100 Received: from CAPPRWMEHC02.central.ad.capita.co.uk (10.154.131.163) by CAPPRWMEEG02.capita.co.uk (10.154.64.242) with Microsoft SMTP Server (TLS) id 14.3.294.0; Fri, 5 Aug 2016 10:24:16 +0100 Received: from edge.capita.co.uk (10.154.64.242) by CAPPRWMEHC02.central.ad.capita.co.uk (10.154.131.163) with Microsoft SMTP Server (TLS) id 14.3.294.0; Fri, 5 Aug 2016 10:23:42 +0100 Received: from EUR02-HE1-obe.outbound.protection.outlook.com (213.199.180.180) by edge.capita.co.uk (10.154.64.242) with Microsoft SMTP Server (TLS) id 14.3.294.0; Fri, 5 Aug 2016 10:24:13 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=capita.onmicrosoft.com; s=selector1-capita-co-uk; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=cQ8jffXL4gUpH2EKs/wUCuU7xiAPjpXAX54+HhPUHH0=; b=Yko4TLnybdWfjIxfjggTtJc2c5uzwQ9A8Z3MFPaolZ+TCys+i2yBp8jlXFy7KPityZ2oLTwo1qZCUY3+goWFnpo5mwOHS1sbSfaanrc9U001DXlx09PYOmfr2Ng22JlsJ23Uy30n0xu7P09qjX8kWde50IRbpwusWLR9XpnPTTs= Received: from AM3PR04MB1202.eurprd04.prod.outlook.com (10.163.6.148) by AM3PR04MB1204.eurprd04.prod.outlook.com (10.163.6.15) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.557.8; Fri, 5 Aug 2016 09:24:01 +0000 Received: from AM3PR04MB1202.eurprd04.prod.outlook.com ([fe80::6197:3592:b365:95ab]) by AM3PR04MB1202.eurprd04.prod.outlook.com ([fe80::6197:3592:b365:95ab%14]) with mapi id 15.01.0557.009; Fri, 5 Aug 2016 09:24:02 +0000 From: "Lavery, Benjamin (IT Enterprise Services)" To: "freebsd-hardware@freebsd.org" Subject: Serial redirection halts on Dell PowerEdge 1955 Blade (mfsBSD) Thread-Topic: Serial redirection halts on Dell PowerEdge 1955 Blade (mfsBSD) Thread-Index: AdHu+A/aAVoMm0dQTau1lXQn8/FjBw== Date: Fri, 5 Aug 2016 09:24:01 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Ben.Lavery@capita.co.uk; x-originating-ip: [82.203.17.6] x-ms-office365-filtering-correlation-id: b93fdc70-ed0b-4a14-db03-08d3bd123d1f x-microsoft-exchange-diagnostics: 1; AM3PR04MB1204; 20:0SwRU2irePnVnAqUI2BO/t/3X4QBh2dCj76/pNYl6DtPTdPj/fjIPd52S6PFmaqj4Vt4upHjpHyhN0LKFc2ryorjCW1eIWnb8NX6dyxuOyTaAMJfP1LpD9tDACJwwcJEF4YghrISH1zRotoQV8Z+lUqpuX6FoTFeNQ/oufsuUEdUAm9C1Fiwm6+MMvmKKbVRzEpJDEHbdV2/V1T2VnkzZOHT3voGHzJQ+9T9hZjMK+Vk8YbKLcETOeRQU6kPTuONuvng2e51jarqhAVa7Y2AZPAYd/d9jsAPzoriFwq8xyt0g8wGPVeHpt4UkoAM8+J5G31ej2voVc5LJazOpl6MzJFYymN0uuV442pr6irdUBEQ7Oj2ChSxCOlAWMjSwLGstaZXGC1uxPwZXdIZYvn8McKrmhluyHQ85qiumHWfwt7+d1FMAHz4mbBb5XPmdTAOPO1BAdX4MI0/lwScGQNqfJXvZNf+fI8T19aY3RIDS6KOpQtD6gmSsLjdUkKk9+uo x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM3PR04MB1204; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(17755550239193); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(102415321)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046); SRVR:AM3PR04MB1204; BCL:0; PCL:0; RULEID:; SRVR:AM3PR04MB1204; x-forefront-prvs: 0025434D2D x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(979002)(6009001)(7916002)(66654002)(199003)(189002)(3280700002)(74482002)(5002640100001)(9686002)(86362001)(33656002)(3660700001)(5890100001)(105586002)(81156014)(19580405001)(450100001)(2501003)(5250100002)(74316002)(5640700001)(305945005)(92566002)(87936001)(8936002)(102836003)(229853001)(2351001)(107886002)(68736007)(66066001)(2906002)(189998001)(81166006)(7846002)(97736004)(8676002)(50986999)(2900100001)(110136002)(19580395003)(99936001)(101416001)(3846002)(7696003)(586003)(6116002)(76576001)(106356001)(54356999)(7736002)(357404004)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1102; SCL:1; SRVR:AM3PR04MB1204; H:AM3PR04MB1202.eurprd04.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:; received-spf: None (protection.outlook.com: capita.co.uk does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Aug 2016 09:24:01.9768 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 1edaad83-b2ef-483d-81f1-2c48682f40ec X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR04MB1204 X-ExSBR-Organization: edge.capita.co.uk X-OriginatorOrg: capita.co.uk X-Scanned-By: MailControl 44278.1378 (www.mailcontrol.com) on 10.65.0.139 Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Aug 2016 10:32:50 -0000 Good morning all, I have been given an old Dell PowerEdge chassis filled with PowerEdge 1955 = blades. I also have a FreeNAS which I'm using to serve DHCP addresses and = TFTP for PXE booting. I can get the blades to PXE boot (I'm using pxelinux to load mfsbsd-10.3-RE= LEASE-amd64), but the serial redirection is only working for the first part= of the boot sequence. After select mfsBSD at the menu, I get the followin= g (verbose boot chosen at the boot menu): Loading memdisk... ok ok e820: 0000000000000000 00000000000a0000 1 e820: 0000000000100000 00000000cfea8000 1 e820: 00000000cffa8000 000000000000fc00 3 e820: 00000000cffb7c00 0000000000048400 2 Total size needed =3D 2513 bytes, allocating 3K Old dos memory at 0x8a000 (map says 0xa0000), loading at 0x89400 1588: 0xffff 15E801: 0x3c00 0xcbba INT 13 08: Success, count =3D 1, BPT =3D 0000:0000 Drive probing gives drive shift limit: 0x82 old: int13 =3D ca001da6 int15 =3D f000f859 int1e =3D f000efc7 new: int13 =3D 8940000a int15 =3D 894003ba int1e =3D f000efc7 Loading boot sector... booting... /boot/config: -D =20=20=20=20 FreeBSD/x86 boot Default: 0:ad(0p2)/boot/loader boot: BTX loader 1.00 BTX version is 1.02 console comconsole failed to initialize Consoles: internal video/keyboard serial port BIOS drive A: is disk0 BIOS drive C: is disk1 BIOS drive D: is disk2 BIOS 549kB/3353216kB available memory =20=20=20=20 FreeBSD/x86 bootstrap loader, Revision 1.1 (root@releng1.nyi.freebsd.org, Fri Mar 25 02:04:38 UTC 2016) \oading /boot/defaults/loader.conf lapic3: Routing NMI -> LINT1 lapic3: LINT1 trigger: edge lapic3: LINT1 polarity: high lapic7: Routing NMI -> LINT1 lapic7: LINT1 trigger: edge lapic7: LINT1 polarity: high MADT: Interrupt override: source 0, irq 2 ioapic0: Routing IRQ 0 -> intpin 2 MADT: Interrupt override: source 9, irq 9 ioapic0: intpin 9 trigger: level ioapic0 irqs 0-23 on motherboard ioapic1 irqs 32-55 on motherboard ioapic2 irqs 192-215 on motherboard cpu0 BSP: ID: 0x00000000 VER: 0x00050014 LDR: 0x00000000 DFR: 0xffffffff lint0: 0x00010700 lint1: 0x00000400 TPR: 0x00000000 SVR: 0x000001ff timer: 0x000100ef therm: 0x00010000 err: 0x000000f0 pmc: 0x00010400 wlan: <802.11 Link Layer> snd_unit_init() u=3D0x00ff8000 [512] d=3D0x00007c00 [32] c=3D0x000003ff= [1024] feeder_register: snd_unit=3D-1 snd_maxautovchans=3D16 latency=3D5 feede= r_rate_min=3D1 fe eder_rate_max=3D2016000 feeder_rate_round=3D25 null: nfslock: pseudo-device =91=91=91=91=91=91=91=91=91=91=91 The odd glitch (=91) is me trying to input after output stops. I'm accessi= ng the console by TELNETing to the blade chassis, and connecting to the bla= de console. I believe this runs at a baud rate of 57600. Because of this = I've modified the following files within mfsBSD: etc/ttys (from inside mfsroot.gz) # Serial terminals ttyu0 "/usr/libexec/getty std.57600" vt100 on secure ttyu1 "/usr/libexec/getty std.57600" dialup on secure ttyu2 "/usr/libexec/getty std.57600" vt100 off secure ttyu3 "/usr/libexec/getty std.57600" vt100 off secure # Dumb console dcons "/usr/libexec/getty std.9600" vt100 off secure boot/loader.conf boot_multicons=3D"YES" boot_serial=3D"YES" comconsole_speed=3D"57600" console=3D"comconsole" The Java console from the chassis web interface doesn't work on my laptop, = but via another server I've been able to grab the attached screenshot. It = states: getty[706]: open /dev/ttyu0: No such file or directory Unfortunately the Java console doesn't work well enough for me to use to lo= g in and get at any settings (getting it to load that far took 10 minutes..= .) The blades BIOS is set up to redirect to COM2 (it's either that or be disab= led) and to emulate vt100/vt200 (could also be ANSI, but this doesn't seem = to allow input). Would anyone have any ideas about how I can get console redirection working= past this point? Or where I can look to get more information? Many thanks, Ben This email is security checked and subject to the disclaimer on web-page: h= ttp://www.capita.co.uk/email-disclaimer.aspx