Date: Mon, 1 Aug 2016 11:48:30 +0200 From: Borja Marcos <borjam@sarenet.es> To: "O. Hartmann" <ohartman@zedat.fu-berlin.de> Cc: Jason Zhang <jasonzhang@cyphytech.com>, freebsd-performance@freebsd.org, freebsd-current@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i Message-ID: <1519EC23-0DBC-4139-96F6-250EF872A14B@sarenet.es> In-Reply-To: <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> <B2239257-E136-4918-8FA3-1EE51BC50FB4@sarenet.es> <20160801084504.563c79cf@freyja.zeit4.iv.bundesimmobilien.de>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 01 Aug 2016, at 08:45, O. Hartmann <ohartman@zedat.fu-berlin.de> =
wrote:
>=20
> On Wed, 22 Jun 2016 08:58:08 +0200
> Borja Marcos <borjam@sarenet.es> wrote:
>=20
>> There is an option you can use (I do it all the time!) to make the =
card
>> behave as a plain HBA so that the disks are handled by the =E2=80=9Cda=E2=
=80=9D driver.=20
>>=20
>> Add this to /boot/loader.conf
>>=20
>> hw.mfi.allow_cam_disk_passthrough=3D1
>> mfip_load=3D=E2=80=9CYES"
>>=20
>> And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To =
avoid confusions, it=E2=80=99s
>> better to make sure the disks are not part of a =E2=80=9Cjbod=E2=80=9D =
or logical volume
>> configuration.
>>=20
>>=20
>>=20
>>=20
>> Borja.
> [...]
>=20
> How is this supposed to work when ALL disks (including boot device) =
are settled
> with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 =
and
> detected within FreeBSD 11-BETA and 12-CURRENT) controller itself?
>=20
> I did not find any solution to force the CP400i into a mode making =
itself
> acting as a HBA (we intend to use all drives with ZFS and let FreeBSD
> kernel/ZFS control everything).
Have you tried that particular option?=20
With kinda recent LSI based cards you have three options:
- The most usual and definitely NOT RECOMMENDED option is to define a =
logical volume per disk
which actually LSI Logic called before JBOD mode. It=E2=80=99s not =
recommended at all if you want to run ZFS.
- Recent cards, I think I saw this first on the LSI3008, have a JBOD =
mode that exposes the drives as =E2=80=9Cmfisyspd=E2=80=9D devices.
I don=E2=80=99t recommend it either, because the syspd drives are a sort =
of limited version of a disk device. With SSDs, especially, you
don=E2=80=99t have access to the TRIM command.
- The third option is to make the driver expose the SAS devices like a =
HBA would do, so that they are visible to the
CAM layer, and disks are handled by the stock =E2=80=9Cda=E2=80=9D =
driver, which is the ideal solution.=20
However, this third option might not be available in some custom =
firmware versions for certain manufacturers? I don=C2=B4t
know. And I would hesitate to make the conversion on a production =
machine unless you have a complete and reliable
full backup of all the data in case you need to rebuild it.
In order to do it you need a couple of things. You need to set the =
variable hw.mfi.allow_cam_disk_passthrough=3D1
and to load the mfip.ko module.
When booting installation media, enter command mode and use these =
commands:
-----
set hw.mfi.allow_cam_disk_passthrough=3D1
load mfip
boot
=E2=80=94=E2=80=94=E2=80=94
Remember that after installation you need to update /boot/loader.conf in =
the system you just installed with the
following contents:
hw.mfi.allow_cam_disk_passthrough=3D1
mfip_load=3D=E2=80=9CYES=E2=80=9D
A note regarding CAM and MFI visibility: On some old firmware versions =
for the LSI2008 I=E2=80=99ve even seen the disks
available both as =E2=80=9Cmfi=E2=80=9D and =E2=80=9Cda=E2=80=9D =
drivers. If possible, you should try to set them up as =E2=80=9Cunconfigur=
ed good=E2=80=9D on the RAID
firmware. Use the RAID firmware set up or maybe mfiutil(8)
Also, make sure you don=E2=80=99t create any logical volumes on the =
disks you want exposed to CAM. You should delete
the logical volumes so that the MFI firmware doesn=E2=80=99t do anything =
with them.=20
AND BEWARE: Doing these changes to a system in production with valuable =
data is dangerous. Make sure you have a full
and sound backup before making these changes.
As a worst case, the card could expose the devices both as =E2=80=9Csyspd=E2=
=80=9D and CAM (i.e., =E2=80=9Cda=E2=80=9D drives) but as long as you =
don=E2=80=99t
touch the syspd devices the card won=E2=80=99t do anything to them as =
far as I know. It could be a serious problem, however, if you=20
access a drive part of a logical volume through CAM, as RAID cards tend =
do to =E2=80=9Cpatrol reads=E2=80=9D and other stuff on them.=20
Provided it=E2=80=99s safe to do what I recommended, try it and follow =
up by email.=20
Borja.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1519EC23-0DBC-4139-96F6-250EF872A14B>