From owner-freebsd-scsi@FreeBSD.ORG Tue Jan 22 19:14:46 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9A47BB70; Tue, 22 Jan 2013 19:14:46 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog130.obsmtp.com (na3sys009aog130.obsmtp.com [74.125.149.143]) by mx1.freebsd.org (Postfix) with ESMTP id C8DD7EC5; Tue, 22 Jan 2013 19:14:45 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob130.postini.com ([74.125.148.12]) with SMTP ID DSNKUP7lH+fOJ5yDF+0183YVuSdSkPV9kOW6@postini.com; Tue, 22 Jan 2013 11:14:46 PST Received: from PALHUB01.lsi.com (128.94.213.114) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.264.0; Tue, 22 Jan 2013 14:14:33 -0500 Received: from PALEXCH11.lsi.com (128.94.223.42) by PALHUB01.lsi.com (128.94.213.114) with Microsoft SMTP Server (TLS) id 8.3.264.0; Tue, 22 Jan 2013 14:14:38 -0500 Received: from inbexch02.lsi.com (135.36.98.40) by PALEXCH11.lsi.com (128.94.223.42) with Microsoft SMTP Server (TLS) id 14.2.309.2; Tue, 22 Jan 2013 14:14:37 -0500 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch02.lsi.com ([135.36.98.40]) with mapi; Wed, 23 Jan 2013 00:44:34 +0530 From: "Desai, Kashyap" To: "Kenneth D. Merry" Date: Wed, 23 Jan 2013 00:44:31 +0530 Subject: RE: Max Queue depth of HBA limited to 256 ? Thread-Topic: Max Queue depth of HBA limited to 256 ? Thread-Index: Ac33+YtDXsJz1U+4Ry62mJByV4SrKgABRkhgADVxMNA= Message-ID: References: <20130121170529.GA64188@nargothrond.kdm.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-scsi@freebsd.org" , "McConnell, Stephen" , "jhb@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2013 19:14:46 -0000 LSI h/w needs more outstanding command in FW to get better Perf counts comp= are to other OS. Please suggest if whatever I have been observed is limitation from FreeBSD = or we can tune it in Driver ? My goals is to pump ~1000 outstanding IOs to the HBA. I see that it never g= oes beyond 255.=20 Thanks, Kashyap > -----Original Message----- > From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- > scsi@freebsd.org] On Behalf Of Desai, Kashyap > Sent: Monday, January 21, 2013 11:18 PM > To: Kenneth D. Merry > Cc: freebsd-scsi@freebsd.org; jhb@freebsd.org; McConnell, Stephen > Subject: RE: Max Queue depth of HBA limited to 256 ? >=20 >=20 >=20 > > -----Original Message----- > > From: Kenneth D. Merry [mailto:ken@freebsd.org] > > Sent: Monday, January 21, 2013 10:35 PM > > To: Desai, Kashyap > > Cc: freebsd-scsi@freebsd.org; McConnell, Stephen; Saxena, Sumit; > > jhb@freebsd.org > > Subject: Re: Max Queue depth of HBA limited to 256 ? > > > > On Mon, Jan 21, 2013 at 20:15:47 +0530, Desai, Kashyap wrote: > > > Hi, > > > > > > I was trying to check few things on LSI controller, where we have > > > more > > than 256 queue depth support. > > > I added default maxtags in scsi/scsi_xpt.c as below. (Because I > > > don't > > want mattags to restrict any outstanding commands the LSI HBA. > > > > > > { > > > /* Default tagged queuing parameters for all devices */ > > > { > > > T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED, > > > /*vendor*/"*", /*product*/"*", /*revision*/"*" > > > }, > > > /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <--- Default > > maxtags were 256. I increase it to 10234 > > > }, > > > > > > > > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding > > commands in Firmware. But due to some reason, I am not able to pump > > more than 256 outstanding commands to the HBA. > > > > > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have > > sysctl parameter in Driver to display outstanding "FW commands". Max > > value for FW outstanding only goes up to 256. > > > > > > Also from some other mail thread Subject "mfi driver performance", I > > found that folks talk about tuning queue depth _but_ nobody discussed > > to increase it beyond 256. Is there any limitation in FreeBSD ? > > > > > > > As Jim pointed out, one thing to check is the values passed into > > cam_sim_alloc(). In the case of the mps(4) driver, the calculation is > > in mps_attach(): > > > > sc->num_reqs =3D MIN(MPS_REQ_FRAMES, sc->facts->RequestCredit); > > > > What is reported for the RequestCredit on this particular adapter? > > > > The other question is, what does 'camcontrol tags daX -v' show when > > you are running the test? >=20 > Below is output of camcontrol tags da1 -v. >=20 > dhcp-135-24-192-127# camcontrol tags da13 -v > (pass13:mrsas0:0:13:0): dev_openings 1024 > (pass13:mrsas0:0:13:0): dev_active 0 > (pass13:mrsas0:0:13:0): devq_openings 1024 > (pass13:mrsas0:0:13:0): devq_queued 0 > (pass13:mrsas0:0:13:0): held 0 > (pass13:mrsas0:0:13:0): mintags 2 > (pass13:mrsas0:0:13:0): maxtags 1024 > dhcp-135-24-192-127# camcontrol tags da1 -v > (pass1:mrsas0:0:1:0): dev_openings 1024 > (pass1:mrsas0:0:1:0): dev_active 0 > (pass1:mrsas0:0:1:0): devq_openings 1024 > (pass1:mrsas0:0:1:0): devq_queued 0 > (pass1:mrsas0:0:1:0): held 0 > (pass1:mrsas0:0:1:0): mintags 2 > (pass1:mrsas0:0:1:0): maxtags 1024 >=20 > Value 1024 is hard coded for my testing. In MegaRaid controller and SAS- > HBA Driver read max commands value from FW. > Similar to "RequestCredit".. Different FW has different value, but they > are every time above 255. >=20 >=20 > When I run IOs dev_active stays in range of 0-255 only. See below > output when I run IOs on /dev/da1 and /dev/da13. I expect total > dev_openings should go beyond 255, which is not happening. >=20 >=20 > dhcp-135-24-192-127# camcontrol tags da1 -v > (pass1:mrsas0:0:1:0): dev_openings 832 > (pass1:mrsas0:0:1:0): dev_active 192 > (pass1:mrsas0:0:1:0): devq_openings 832 > (pass1:mrsas0:0:1:0): devq_queued 0 > (pass1:mrsas0:0:1:0): held 0 > (pass1:mrsas0:0:1:0): mintags 2 > (pass1:mrsas0:0:1:0): maxtags 1024 > dhcp-135-24-192-127# camcontrol tags da13 -v > (pass13:mrsas0:0:13:0): dev_openings 881 > (pass13:mrsas0:0:13:0): dev_active 143 > (pass13:mrsas0:0:13:0): devq_openings 881 > (pass13:mrsas0:0:13:0): devq_queued 0 > (pass13:mrsas0:0:13:0): held 0 > (pass13:mrsas0:0:13:0): mintags 2 > (pass13:mrsas0:0:13:0): maxtags 1024 >=20 >=20 >=20 >=20 > Jim: > Below is my API call. I have hard code value "queue_depth" =3D 1024 >=20 > sc->sim_0 =3D cam_sim_alloc(mrsas_action, mrsas_poll, "mrsas", sc, > device_get_unit(sc->mrsas_dev), &sc->sim_lock, queue_depth, > queue_depth, devq); >=20 > ~ Kashyap >=20 > > > > Ken > > -- > > Kenneth Merry > > ken@FreeBSD.ORG > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"