From owner-freebsd-stable@freebsd.org Mon Aug 1 09:56:44 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD56EBAA2C5; Mon, 1 Aug 2016 09:56:44 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3AE8E1812; Mon, 1 Aug 2016 09:56:43 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id 5B5319DC575; Mon, 1 Aug 2016 11:48:31 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Borja Marcos In-Reply-To: <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> Date: Mon, 1 Aug 2016 11:48:30 +0200 Cc: Jason Zhang , freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> To: "O. Hartmann" X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 09:56:44 -0000 > On 01 Aug 2016, at 08:45, O. Hartmann = wrote: >=20 > On Wed, 22 Jun 2016 08:58:08 +0200 > Borja Marcos wrote: >=20 >> There is an option you can use (I do it all the time!) to make the = card >> behave as a plain HBA so that the disks are handled by the =E2=80=9Cda=E2= =80=9D driver.=20 >>=20 >> Add this to /boot/loader.conf >>=20 >> hw.mfi.allow_cam_disk_passthrough=3D1 >> mfip_load=3D=E2=80=9CYES" >>=20 >> And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To = avoid confusions, it=E2=80=99s >> better to make sure the disks are not part of a =E2=80=9Cjbod=E2=80=9D = or logical volume >> configuration. >>=20 >>=20 >>=20 >>=20 >> Borja. > [...] >=20 > How is this supposed to work when ALL disks (including boot device) = are settled > with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 = and > detected within FreeBSD 11-BETA and 12-CURRENT) controller itself? >=20 > I did not find any solution to force the CP400i into a mode making = itself > acting as a HBA (we intend to use all drives with ZFS and let FreeBSD > kernel/ZFS control everything). Have you tried that particular option?=20 With kinda recent LSI based cards you have three options: - The most usual and definitely NOT RECOMMENDED option is to define a = logical volume per disk which actually LSI Logic called before JBOD mode. It=E2=80=99s not = recommended at all if you want to run ZFS. - Recent cards, I think I saw this first on the LSI3008, have a JBOD = mode that exposes the drives as =E2=80=9Cmfisyspd=E2=80=9D devices. I don=E2=80=99t recommend it either, because the syspd drives are a sort = of limited version of a disk device. With SSDs, especially, you don=E2=80=99t have access to the TRIM command. - The third option is to make the driver expose the SAS devices like a = HBA would do, so that they are visible to the CAM layer, and disks are handled by the stock =E2=80=9Cda=E2=80=9D = driver, which is the ideal solution.=20 However, this third option might not be available in some custom = firmware versions for certain manufacturers? I don=C2=B4t know. And I would hesitate to make the conversion on a production = machine unless you have a complete and reliable full backup of all the data in case you need to rebuild it. In order to do it you need a couple of things. You need to set the = variable hw.mfi.allow_cam_disk_passthrough=3D1 and to load the mfip.ko module. When booting installation media, enter command mode and use these = commands: ----- set hw.mfi.allow_cam_disk_passthrough=3D1 load mfip boot =E2=80=94=E2=80=94=E2=80=94 Remember that after installation you need to update /boot/loader.conf in = the system you just installed with the following contents: hw.mfi.allow_cam_disk_passthrough=3D1 mfip_load=3D=E2=80=9CYES=E2=80=9D A note regarding CAM and MFI visibility: On some old firmware versions = for the LSI2008 I=E2=80=99ve even seen the disks available both as =E2=80=9Cmfi=E2=80=9D and =E2=80=9Cda=E2=80=9D = drivers. If possible, you should try to set them up as =E2=80=9Cunconfigur= ed good=E2=80=9D on the RAID firmware. Use the RAID firmware set up or maybe mfiutil(8) Also, make sure you don=E2=80=99t create any logical volumes on the = disks you want exposed to CAM. You should delete the logical volumes so that the MFI firmware doesn=E2=80=99t do anything = with them.=20 AND BEWARE: Doing these changes to a system in production with valuable = data is dangerous. Make sure you have a full and sound backup before making these changes. As a worst case, the card could expose the devices both as =E2=80=9Csyspd=E2= =80=9D and CAM (i.e., =E2=80=9Cda=E2=80=9D drives) but as long as you = don=E2=80=99t touch the syspd devices the card won=E2=80=99t do anything to them as = far as I know. It could be a serious problem, however, if you=20 access a drive part of a logical volume through CAM, as RAID cards tend = do to =E2=80=9Cpatrol reads=E2=80=9D and other stuff on them.=20 Provided it=E2=80=99s safe to do what I recommended, try it and follow = up by email.=20 Borja.