Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 23 Jan 2013 00:44:31 +0530
From:      "Desai, Kashyap" <Kashyap.Desai@lsi.com>
To:        "Kenneth D. Merry" <ken@freebsd.org>
Cc:        "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org>, "McConnell, Stephen" <Stephen.McConnell@lsi.com>, "jhb@freebsd.org" <jhb@freebsd.org>
Subject:   RE: Max Queue depth of HBA limited to 256 ?
Message-ID:  <B2FD678A64EAAD45B089B123FDFC3ED75E68FD6296@inbmail01.lsi.com>
In-Reply-To: <B2FD678A64EAAD45B089B123FDFC3ED75E68FD616E@inbmail01.lsi.com>
References:  <B2FD678A64EAAD45B089B123FDFC3ED75E68FD6158@inbmail01.lsi.com> <20130121170529.GA64188@nargothrond.kdm.org> <B2FD678A64EAAD45B089B123FDFC3ED75E68FD616E@inbmail01.lsi.com>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
LSI h/w needs more outstanding command in FW to get better Perf counts comp=
are to other OS.

Please suggest if whatever I have been observed is limitation from FreeBSD =
or we can tune it in Driver ?
My goals is to pump ~1000 outstanding IOs to the HBA. I see that it never g=
oes beyond 255.=20

Thanks,
Kashyap

> -----Original Message-----
> From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd-
> scsi@freebsd.org] On Behalf Of Desai, Kashyap
> Sent: Monday, January 21, 2013 11:18 PM
> To: Kenneth D. Merry
> Cc: freebsd-scsi@freebsd.org; jhb@freebsd.org; McConnell, Stephen
> Subject: RE: Max Queue depth of HBA limited to 256 ?
>=20
>=20
>=20
> > -----Original Message-----
> > From: Kenneth D. Merry [mailto:ken@freebsd.org]
> > Sent: Monday, January 21, 2013 10:35 PM
> > To: Desai, Kashyap
> > Cc: freebsd-scsi@freebsd.org; McConnell, Stephen; Saxena, Sumit;
> > jhb@freebsd.org
> > Subject: Re: Max Queue depth of HBA limited to 256 ?
> >
> > On Mon, Jan 21, 2013 at 20:15:47 +0530, Desai, Kashyap wrote:
> > > Hi,
> > >
> > > I was trying to check few things on LSI controller, where we have
> > > more
> > than 256 queue depth support.
> > > I added default maxtags in scsi/scsi_xpt.c as below. (Because I
> > > don't
> > want mattags to restrict any outstanding commands the LSI HBA.
> > >
> > >     {
> > >         /* Default tagged queuing parameters for all devices */
> > >         {
> > >           T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED,
> > >           /*vendor*/"*", /*product*/"*", /*revision*/"*"
> > >         },
> > >         /*quirks*/0, /*mintags*/2, /*maxtags*/1024      <--- Default
> > maxtags were 256. I increase it to 10234
> > >     },
> > >
> > >
> > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding
> > commands in Firmware.  But due to some reason, I am not able to pump
> > more than 256 outstanding commands to the HBA.
> > >
> > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have
> > sysctl parameter in Driver to display outstanding "FW commands". Max
> > value for FW outstanding only goes up to 256.
> > >
> > > Also from some other mail thread Subject "mfi driver performance", I
> > found that folks talk about tuning queue depth _but_ nobody discussed
> > to increase it beyond 256. Is there any limitation in FreeBSD ?
> > >
> >
> > As Jim pointed out, one thing to check is the values passed into
> > cam_sim_alloc().  In the case of the mps(4) driver, the calculation is
> > in mps_attach():
> >
> > sc->num_reqs =3D MIN(MPS_REQ_FRAMES, sc->facts->RequestCredit);
> >
> > What is reported for the RequestCredit on this particular adapter?
> >
> > The other question is, what does 'camcontrol tags daX -v' show when
> > you are running the test?
>=20
> Below is output of camcontrol tags da1 -v.
>=20
> dhcp-135-24-192-127# camcontrol tags da13 -v
> (pass13:mrsas0:0:13:0): dev_openings  1024
> (pass13:mrsas0:0:13:0): dev_active    0
> (pass13:mrsas0:0:13:0): devq_openings 1024
> (pass13:mrsas0:0:13:0): devq_queued   0
> (pass13:mrsas0:0:13:0): held          0
> (pass13:mrsas0:0:13:0): mintags       2
> (pass13:mrsas0:0:13:0): maxtags       1024
> dhcp-135-24-192-127# camcontrol tags da1 -v
> (pass1:mrsas0:0:1:0): dev_openings  1024
> (pass1:mrsas0:0:1:0): dev_active    0
> (pass1:mrsas0:0:1:0): devq_openings 1024
> (pass1:mrsas0:0:1:0): devq_queued   0
> (pass1:mrsas0:0:1:0): held          0
> (pass1:mrsas0:0:1:0): mintags       2
> (pass1:mrsas0:0:1:0): maxtags       1024
>=20
> Value 1024 is hard coded for my testing. In MegaRaid controller and SAS-
> HBA Driver read max commands value from FW.
> Similar to "RequestCredit"..  Different FW has different value, but they
> are every time above 255.
>=20
>=20
> When I run IOs dev_active stays in range of 0-255 only.  See below
> output when I run IOs on /dev/da1 and /dev/da13. I expect total
> dev_openings should go beyond 255, which is not happening.
>=20
>=20
> dhcp-135-24-192-127# camcontrol tags da1 -v
> (pass1:mrsas0:0:1:0): dev_openings  832
> (pass1:mrsas0:0:1:0): dev_active    192
> (pass1:mrsas0:0:1:0): devq_openings 832
> (pass1:mrsas0:0:1:0): devq_queued   0
> (pass1:mrsas0:0:1:0): held          0
> (pass1:mrsas0:0:1:0): mintags       2
> (pass1:mrsas0:0:1:0): maxtags       1024
> dhcp-135-24-192-127# camcontrol tags da13 -v
> (pass13:mrsas0:0:13:0): dev_openings  881
> (pass13:mrsas0:0:13:0): dev_active    143
> (pass13:mrsas0:0:13:0): devq_openings 881
> (pass13:mrsas0:0:13:0): devq_queued   0
> (pass13:mrsas0:0:13:0): held          0
> (pass13:mrsas0:0:13:0): mintags       2
> (pass13:mrsas0:0:13:0): maxtags       1024
>=20
>=20
>=20
>=20
> Jim:
> Below is my API call. I have hard code value "queue_depth" =3D 1024
>=20
>     sc->sim_0 =3D cam_sim_alloc(mrsas_action, mrsas_poll, "mrsas", sc,
>         device_get_unit(sc->mrsas_dev), &sc->sim_lock, queue_depth,
>         queue_depth, devq);
>=20
> ~ Kashyap
>=20
> >
> > Ken
> > --
> > Kenneth Merry
> > ken@FreeBSD.ORG
> _______________________________________________
> freebsd-scsi@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-scsi
> To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"



Want to link to this message? Use this URL: <http://docs.FreeBSD.org/cgi/mid.cgi?B2FD678A64EAAD45B089B123FDFC3ED75E68FD6296>