From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 21 11:06:52 2013 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 83B61CB5 for ; Mon, 21 Jan 2013 11:06:52 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 7600C731 for ; Mon, 21 Jan 2013 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r0LB6qFZ054209 for ; Mon, 21 Jan 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r0LB6qAs054207 for freebsd-scsi@FreeBSD.org; Mon, 21 Jan 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 21 Jan 2013 11:06:52 GMT Message-Id: <201301211106.r0LB6qAs054207@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Subject: Current problem reports assigned to freebsd-scsi@FreeBSD.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jan 2013 11:06:52 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/171650 scsi [da] da(4) driver does not recognize end of cciss (Sma o kern/169403 scsi [cam] [patch] CAM layer, I/O starvation, no fairness o kern/165982 scsi [mpt] mpt instability, drive resets, and losses on Fre o kern/165740 scsi [cam] SCSI code must drain callbacks before free o kern/163713 scsi [aic7xxx] [patch] Add Adaptec29329LPE to aic79xx_pci.c o kern/162256 scsi [mpt] QUEUE FULL EVENT and 'mpt_cam_event: 0x0' o kern/161809 scsi [cam] [patch] set kern.cam.boot_delay via build option o kern/157770 scsi [iscsi] [panic] iscsi_initiator panic o kern/154432 scsi [xpt] run_interrupt_driven_hooks: still waiting after o kern/153514 scsi [cam] [panic] CAM related panic o docs/151336 scsi Missing documentation of scsi_ and ata_ functions in c s kern/149927 scsi [cam] hard drive not stopped before removing power dur o kern/148083 scsi [aac] Strange device reporting o kern/147704 scsi [mpt] sys/dev/mpt: new chip revision, partially unsupp o kern/145768 scsi [mpt] can't perform I/O on SAS based SAN disk in freeb o kern/144648 scsi [aac] Strange values of speed and bus width in dmesg o kern/142351 scsi [mpt] LSILogic driver performance problems o kern/134488 scsi [mpt] MPT SCSI driver probes max. 8 LUNs per device o kern/132206 scsi [mpt] system panics on boot when mirroring and 2nd dri o kern/130621 scsi [mpt] tranfer rate is inscrutable slow when use lsi213 o kern/129602 scsi [ahd] ahd(4) gets confused and wedges SCSI bus o kern/128452 scsi [sa] [panic] Accessing SCSI tape drive randomly crashe o kern/128245 scsi [scsi] "inquiry data fails comparison at DV1 step" [re o kern/127927 scsi [isp] isp(4) target driver crashes kernel when set up o kern/127717 scsi [ata] [patch] [request] - support write cache toggling o kern/123674 scsi [ahc] ahc driver dumping o kern/123520 scsi [ahd] unable to boot from net while using ahd o sparc/121676 scsi [iscsi] iscontrol do not connect iscsi-target on sparc o kern/120487 scsi [sg] scsi_sg incompatible with scanners o kern/120247 scsi [mpt] FreeBSD 6.3 and LSI Logic 1030 = only 3.300MB/s o kern/114597 scsi [sym] System hangs at SCSI bus reset with dual HBAs o kern/110847 scsi [ahd] Tyan U320 onboard problem with more than 3 disks o kern/99954 scsi [ahc] reading from DVD failes on 6.x [regression] o kern/92798 scsi [ahc] SCSI problem with timeouts o kern/90282 scsi [sym] SCSI bus resets cause loss of ch device o kern/76178 scsi [ahd] Problem with ahd and large SCSI Raid system o kern/74627 scsi [ahc] [hang] Adaptec 2940U2W Can't boot 5.3 s kern/61165 scsi [panic] kernel page fault after calling cam_send_ccb o kern/60641 scsi [sym] Sporadic SCSI bus resets with 53C810 under load o kern/60598 scsi wire down of scsi devices conflicts with config s kern/57398 scsi [mly] Current fails to install on mly(4) based RAID di o kern/52638 scsi [panic] SCSI U320 on SMP server won't run faster than o kern/44587 scsi dev/dpt/dpt.h is missing defines required for DPT_HAND o kern/39388 scsi ncr/sym drivers fail with 53c810 and more than 256MB m o kern/35234 scsi World access to /dev/pass? (for scanner) requires acce 45 problems total. From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 21 14:46:10 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2C273D54; Mon, 21 Jan 2013 14:46:10 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog138.obsmtp.com (na3sys009aog138.obsmtp.com [74.125.149.19]) by mx1.freebsd.org (Postfix) with ESMTP id 7B0706FF; Mon, 21 Jan 2013 14:46:09 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob138.postini.com ([74.125.148.12]) with SMTP ID DSNKUP1UoF9foAMAHcbawBf6sEDcCfHXb1xI@postini.com; Mon, 21 Jan 2013 06:46:09 PST Received: from PALCAS01.lsi.com (128.94.213.117) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.264.0; Mon, 21 Jan 2013 09:45:49 -0500 Received: from inbexch01.lsi.com (135.36.98.37) by PALCAS01.lsi.com (128.94.213.117) with Microsoft SMTP Server (TLS) id 8.3.264.0; Mon, 21 Jan 2013 09:45:52 -0500 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch01.lsi.com ([135.36.98.37]) with mapi; Mon, 21 Jan 2013 20:15:48 +0530 From: "Desai, Kashyap" To: "freebsd-scsi@freebsd.org" Date: Mon, 21 Jan 2013 20:15:47 +0530 Subject: Max Queue depth of HBA limited to 256 ? Thread-Topic: Max Queue depth of HBA limited to 256 ? Thread-Index: Ac335f/Ukr+NszdkQQqQECSOu+WvGg== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "Kenneth D. Merry" , "jhb@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jan 2013 14:46:10 -0000 Hi, I was trying to check few things on LSI controller, where we have more than= 256 queue depth support. I added default maxtags in scsi/scsi_xpt.c as below. (Because I don't want = mattags to restrict any outstanding commands the LSI HBA. { /* Default tagged queuing parameters for all devices */ { T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED, /*vendor*/"*", /*product*/"*", /*revision*/"*" }, /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <--- Default maxtag= s were 256. I increase it to 10234 }, LSI's SAS-HBA and MR-HBA can support more than 256 outstanding commands in = Firmware. But due to some reason, I am not able to pump more than 256 outs= tanding commands to the HBA. I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have sysctl par= ameter in Driver to display outstanding "FW commands". Max value for FW out= standing only goes up to 256. Also from some other mail thread Subject "mfi driver performance", I found = that folks talk about tuning queue depth _but_ nobody discussed to increase= it beyond 256. Is there any limitation in FreeBSD ? ~ Kashyap From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 21 16:43:49 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CCE06D5D; Mon, 21 Jan 2013 16:43:49 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) by mx1.freebsd.org (Postfix) with ESMTP id 0B979E58; Mon, 21 Jan 2013 16:43:48 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id 12so1194829wgh.1 for ; Mon, 21 Jan 2013 08:43:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=hDGxrA3IiUWB8nLPPsD9JQU+6nVqR2PUyt3jys4ag9I=; b=PsTIafk3oVc3JbEsXXKJ4/dQAVtnP6PDPSE7L3ZvkAJAHLDNyejOPFeU672uzoOVL+ vTObpTlLanrnj1LLC/lssM/uzGg39A8W/G/und0dD+tZygWCYQYVGzil9MqDfwHIUGwS sekLR7CP2Hwr+qNdkkFzxJEMajXO++KlUP0ztRMXfnOe+46iXAPgP5iakX9l8oSasOGm 0Om7Y4R+PD1gQ2qj+3Be8JVJSRBYk+Al3PwoSHwfHDt4ubacnx00lFKAZqsHpmSlQpS/ TkkMzNrQZiC0ovYtUTsobvAqlleMPNihh5SqEEuq4ZZAMWdIBIGZo4X4YyYm3VmGS/Nc 9ThA== MIME-Version: 1.0 X-Received: by 10.180.80.170 with SMTP id s10mr16402966wix.27.1358786628156; Mon, 21 Jan 2013 08:43:48 -0800 (PST) Sender: jim.harris@gmail.com Received: by 10.217.57.4 with HTTP; Mon, 21 Jan 2013 08:43:48 -0800 (PST) In-Reply-To: References: Date: Mon, 21 Jan 2013 09:43:48 -0700 X-Google-Sender-Auth: fAZNJp0lNk_n0cfU5XrqA2GECmU Message-ID: Subject: Re: Max Queue depth of HBA limited to 256 ? From: Jim Harris To: "Desai, Kashyap" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-scsi@freebsd.org" , "Kenneth D. Merry" , "McConnell, Stephen" , "jhb@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jan 2013 16:43:49 -0000 On Mon, Jan 21, 2013 at 7:45 AM, Desai, Kashyap wrote: > Hi, > > I was trying to check few things on LSI controller, where we have more > than 256 queue depth support. > I added default maxtags in scsi/scsi_xpt.c as below. (Because I don't want > mattags to restrict any outstanding commands the LSI HBA. > > { > /* Default tagged queuing parameters for all devices */ > { > T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED, > /*vendor*/"*", /*product*/"*", /*revision*/"*" > }, > /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <--- Default > maxtags were 256. I increase it to 10234 > }, > > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding commands in > Firmware. But due to some reason, I am not able to pump more than 256 > outstanding commands to the HBA. > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have sysctl > parameter in Driver to display outstanding "FW commands". Max value for FW > outstanding only goes up to 256. > > Also from some other mail thread Subject "mfi driver performance", I found > that folks talk about tuning queue depth _but_ nobody discussed to increase > it beyond 256. Is there any limitation in FreeBSD ? > > What is your driver passing to cam_simq_alloc()? This is where you specify the controller's queue depth. -Jim From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 21 17:05:35 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id EC607374; Mon, 21 Jan 2013 17:05:35 +0000 (UTC) (envelope-from ken@kdm.org) Received: from nargothrond.kdm.org (nargothrond.kdm.org [70.56.43.81]) by mx1.freebsd.org (Postfix) with ESMTP id 98781F48; Mon, 21 Jan 2013 17:05:35 +0000 (UTC) Received: from nargothrond.kdm.org (localhost [127.0.0.1]) by nargothrond.kdm.org (8.14.2/8.14.2) with ESMTP id r0LH5TlA064744; Mon, 21 Jan 2013 10:05:29 -0700 (MST) (envelope-from ken@nargothrond.kdm.org) Received: (from ken@localhost) by nargothrond.kdm.org (8.14.2/8.14.2/Submit) id r0LH5TVN064743; Mon, 21 Jan 2013 10:05:29 -0700 (MST) (envelope-from ken) Date: Mon, 21 Jan 2013 10:05:29 -0700 From: "Kenneth D. Merry" To: "Desai, Kashyap" Subject: Re: Max Queue depth of HBA limited to 256 ? Message-ID: <20130121170529.GA64188@nargothrond.kdm.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i Cc: "freebsd-scsi@freebsd.org" , "jhb@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jan 2013 17:05:36 -0000 On Mon, Jan 21, 2013 at 20:15:47 +0530, Desai, Kashyap wrote: > Hi, > > I was trying to check few things on LSI controller, where we have more than 256 queue depth support. > I added default maxtags in scsi/scsi_xpt.c as below. (Because I don't want mattags to restrict any outstanding commands the LSI HBA. > > { > /* Default tagged queuing parameters for all devices */ > { > T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED, > /*vendor*/"*", /*product*/"*", /*revision*/"*" > }, > /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <--- Default maxtags were 256. I increase it to 10234 > }, > > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding commands in Firmware. But due to some reason, I am not able to pump more than 256 outstanding commands to the HBA. > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have sysctl parameter in Driver to display outstanding "FW commands". Max value for FW outstanding only goes up to 256. > > Also from some other mail thread Subject "mfi driver performance", I found that folks talk about tuning queue depth _but_ nobody discussed to increase it beyond 256. Is there any limitation in FreeBSD ? > As Jim pointed out, one thing to check is the values passed into cam_sim_alloc(). In the case of the mps(4) driver, the calculation is in mps_attach(): sc->num_reqs = MIN(MPS_REQ_FRAMES, sc->facts->RequestCredit); What is reported for the RequestCredit on this particular adapter? The other question is, what does 'camcontrol tags daX -v' show when you are running the test? Ken -- Kenneth Merry ken@FreeBSD.ORG From owner-freebsd-scsi@FreeBSD.ORG Mon Jan 21 17:49:46 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 55BC570C; Mon, 21 Jan 2013 17:49:46 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog101.obsmtp.com (na3sys009aog101.obsmtp.com [74.125.149.67]) by mx1.freebsd.org (Postfix) with ESMTP id B3DFE1EE; Mon, 21 Jan 2013 17:49:45 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob101.postini.com ([74.125.148.12]) with SMTP ID DSNKUP1/s5yURRJ2/0o5zYHfzrZKcT6P0e1g@postini.com; Mon, 21 Jan 2013 09:49:46 PST Received: from PALCAS01.lsi.com (128.94.213.117) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.264.0; Mon, 21 Jan 2013 12:48:27 -0500 Received: from inbexch01.lsi.com (135.36.98.37) by PALCAS01.lsi.com (128.94.213.117) with Microsoft SMTP Server (TLS) id 8.3.264.0; Mon, 21 Jan 2013 12:48:29 -0500 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch01.lsi.com ([135.36.98.37]) with mapi; Mon, 21 Jan 2013 23:18:26 +0530 From: "Desai, Kashyap" To: "Kenneth D. Merry" Date: Mon, 21 Jan 2013 23:18:24 +0530 Subject: RE: Max Queue depth of HBA limited to 256 ? Thread-Topic: Max Queue depth of HBA limited to 256 ? Thread-Index: Ac33+YtDXsJz1U+4Ry62mJByV4SrKgABRkhg Message-ID: References: <20130121170529.GA64188@nargothrond.kdm.org> In-Reply-To: <20130121170529.GA64188@nargothrond.kdm.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-scsi@freebsd.org" , "jhb@freebsd.org" , "McConnell, Stephen" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jan 2013 17:49:46 -0000 > -----Original Message----- > From: Kenneth D. Merry [mailto:ken@freebsd.org] > Sent: Monday, January 21, 2013 10:35 PM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; McConnell, Stephen; Saxena, Sumit; > jhb@freebsd.org > Subject: Re: Max Queue depth of HBA limited to 256 ? >=20 > On Mon, Jan 21, 2013 at 20:15:47 +0530, Desai, Kashyap wrote: > > Hi, > > > > I was trying to check few things on LSI controller, where we have more > than 256 queue depth support. > > I added default maxtags in scsi/scsi_xpt.c as below. (Because I don't > want mattags to restrict any outstanding commands the LSI HBA. > > > > { > > /* Default tagged queuing parameters for all devices */ > > { > > T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED, > > /*vendor*/"*", /*product*/"*", /*revision*/"*" > > }, > > /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <--- Default > maxtags were 256. I increase it to 10234 > > }, > > > > > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding > commands in Firmware. But due to some reason, I am not able to pump > more than 256 outstanding commands to the HBA. > > > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have > sysctl parameter in Driver to display outstanding "FW commands". Max > value for FW outstanding only goes up to 256. > > > > Also from some other mail thread Subject "mfi driver performance", I > found that folks talk about tuning queue depth _but_ nobody discussed to > increase it beyond 256. Is there any limitation in FreeBSD ? > > >=20 > As Jim pointed out, one thing to check is the values passed into > cam_sim_alloc(). In the case of the mps(4) driver, the calculation is > in mps_attach(): >=20 > sc->num_reqs =3D MIN(MPS_REQ_FRAMES, sc->facts->RequestCredit); >=20 > What is reported for the RequestCredit on this particular adapter? >=20 > The other question is, what does 'camcontrol tags daX -v' show when you > are running the test? Below is output of camcontrol tags da1 -v. dhcp-135-24-192-127# camcontrol tags da13 -v (pass13:mrsas0:0:13:0): dev_openings 1024 (pass13:mrsas0:0:13:0): dev_active 0 (pass13:mrsas0:0:13:0): devq_openings 1024 (pass13:mrsas0:0:13:0): devq_queued 0 (pass13:mrsas0:0:13:0): held 0 (pass13:mrsas0:0:13:0): mintags 2 (pass13:mrsas0:0:13:0): maxtags 1024 dhcp-135-24-192-127# camcontrol tags da1 -v (pass1:mrsas0:0:1:0): dev_openings 1024 (pass1:mrsas0:0:1:0): dev_active 0 (pass1:mrsas0:0:1:0): devq_openings 1024 (pass1:mrsas0:0:1:0): devq_queued 0 (pass1:mrsas0:0:1:0): held 0 (pass1:mrsas0:0:1:0): mintags 2 (pass1:mrsas0:0:1:0): maxtags 1024 Value 1024 is hard coded for my testing. In MegaRaid controller and SAS-HBA= Driver read max commands value from FW.=20 Similar to "RequestCredit".. Different FW has different value, but they ar= e every time above 255. When I run IOs dev_active stays in range of 0-255 only. See below output w= hen I run IOs on /dev/da1 and /dev/da13. I expect total dev_openings should= go beyond 255, which is not happening. =20 dhcp-135-24-192-127# camcontrol tags da1 -v (pass1:mrsas0:0:1:0): dev_openings 832 (pass1:mrsas0:0:1:0): dev_active 192 (pass1:mrsas0:0:1:0): devq_openings 832 (pass1:mrsas0:0:1:0): devq_queued 0 (pass1:mrsas0:0:1:0): held 0 (pass1:mrsas0:0:1:0): mintags 2 (pass1:mrsas0:0:1:0): maxtags 1024 dhcp-135-24-192-127# camcontrol tags da13 -v (pass13:mrsas0:0:13:0): dev_openings 881 (pass13:mrsas0:0:13:0): dev_active 143 (pass13:mrsas0:0:13:0): devq_openings 881 (pass13:mrsas0:0:13:0): devq_queued 0 (pass13:mrsas0:0:13:0): held 0 (pass13:mrsas0:0:13:0): mintags 2 (pass13:mrsas0:0:13:0): maxtags 1024 Jim: Below is my API call. I have hard code value "queue_depth" =3D 1024=20 sc->sim_0 =3D cam_sim_alloc(mrsas_action, mrsas_poll, "mrsas", sc, device_get_unit(sc->mrsas_dev), &sc->sim_lock, queue_depth, queue_depth, devq); ~ Kashyap >=20 > Ken > -- > Kenneth Merry > ken@FreeBSD.ORG From owner-freebsd-scsi@FreeBSD.ORG Tue Jan 22 19:14:46 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9A47BB70; Tue, 22 Jan 2013 19:14:46 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog130.obsmtp.com (na3sys009aog130.obsmtp.com [74.125.149.143]) by mx1.freebsd.org (Postfix) with ESMTP id C8DD7EC5; Tue, 22 Jan 2013 19:14:45 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob130.postini.com ([74.125.148.12]) with SMTP ID DSNKUP7lH+fOJ5yDF+0183YVuSdSkPV9kOW6@postini.com; Tue, 22 Jan 2013 11:14:46 PST Received: from PALHUB01.lsi.com (128.94.213.114) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.264.0; Tue, 22 Jan 2013 14:14:33 -0500 Received: from PALEXCH11.lsi.com (128.94.223.42) by PALHUB01.lsi.com (128.94.213.114) with Microsoft SMTP Server (TLS) id 8.3.264.0; Tue, 22 Jan 2013 14:14:38 -0500 Received: from inbexch02.lsi.com (135.36.98.40) by PALEXCH11.lsi.com (128.94.223.42) with Microsoft SMTP Server (TLS) id 14.2.309.2; Tue, 22 Jan 2013 14:14:37 -0500 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch02.lsi.com ([135.36.98.40]) with mapi; Wed, 23 Jan 2013 00:44:34 +0530 From: "Desai, Kashyap" To: "Kenneth D. Merry" Date: Wed, 23 Jan 2013 00:44:31 +0530 Subject: RE: Max Queue depth of HBA limited to 256 ? Thread-Topic: Max Queue depth of HBA limited to 256 ? Thread-Index: Ac33+YtDXsJz1U+4Ry62mJByV4SrKgABRkhgADVxMNA= Message-ID: References: <20130121170529.GA64188@nargothrond.kdm.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-scsi@freebsd.org" , "McConnell, Stephen" , "jhb@freebsd.org" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2013 19:14:46 -0000 LSI h/w needs more outstanding command in FW to get better Perf counts comp= are to other OS. Please suggest if whatever I have been observed is limitation from FreeBSD = or we can tune it in Driver ? My goals is to pump ~1000 outstanding IOs to the HBA. I see that it never g= oes beyond 255.=20 Thanks, Kashyap > -----Original Message----- > From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- > scsi@freebsd.org] On Behalf Of Desai, Kashyap > Sent: Monday, January 21, 2013 11:18 PM > To: Kenneth D. Merry > Cc: freebsd-scsi@freebsd.org; jhb@freebsd.org; McConnell, Stephen > Subject: RE: Max Queue depth of HBA limited to 256 ? >=20 >=20 >=20 > > -----Original Message----- > > From: Kenneth D. Merry [mailto:ken@freebsd.org] > > Sent: Monday, January 21, 2013 10:35 PM > > To: Desai, Kashyap > > Cc: freebsd-scsi@freebsd.org; McConnell, Stephen; Saxena, Sumit; > > jhb@freebsd.org > > Subject: Re: Max Queue depth of HBA limited to 256 ? > > > > On Mon, Jan 21, 2013 at 20:15:47 +0530, Desai, Kashyap wrote: > > > Hi, > > > > > > I was trying to check few things on LSI controller, where we have > > > more > > than 256 queue depth support. > > > I added default maxtags in scsi/scsi_xpt.c as below. (Because I > > > don't > > want mattags to restrict any outstanding commands the LSI HBA. > > > > > > { > > > /* Default tagged queuing parameters for all devices */ > > > { > > > T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED, > > > /*vendor*/"*", /*product*/"*", /*revision*/"*" > > > }, > > > /*quirks*/0, /*mintags*/2, /*maxtags*/1024 <--- Default > > maxtags were 256. I increase it to 10234 > > > }, > > > > > > > > > LSI's SAS-HBA and MR-HBA can support more than 256 outstanding > > commands in Firmware. But due to some reason, I am not able to pump > > more than 256 outstanding commands to the HBA. > > > > > > I used "rawio -p 256 /dev/da1" and more /dev/dax in loop. I have > > sysctl parameter in Driver to display outstanding "FW commands". Max > > value for FW outstanding only goes up to 256. > > > > > > Also from some other mail thread Subject "mfi driver performance", I > > found that folks talk about tuning queue depth _but_ nobody discussed > > to increase it beyond 256. Is there any limitation in FreeBSD ? > > > > > > > As Jim pointed out, one thing to check is the values passed into > > cam_sim_alloc(). In the case of the mps(4) driver, the calculation is > > in mps_attach(): > > > > sc->num_reqs =3D MIN(MPS_REQ_FRAMES, sc->facts->RequestCredit); > > > > What is reported for the RequestCredit on this particular adapter? > > > > The other question is, what does 'camcontrol tags daX -v' show when > > you are running the test? >=20 > Below is output of camcontrol tags da1 -v. >=20 > dhcp-135-24-192-127# camcontrol tags da13 -v > (pass13:mrsas0:0:13:0): dev_openings 1024 > (pass13:mrsas0:0:13:0): dev_active 0 > (pass13:mrsas0:0:13:0): devq_openings 1024 > (pass13:mrsas0:0:13:0): devq_queued 0 > (pass13:mrsas0:0:13:0): held 0 > (pass13:mrsas0:0:13:0): mintags 2 > (pass13:mrsas0:0:13:0): maxtags 1024 > dhcp-135-24-192-127# camcontrol tags da1 -v > (pass1:mrsas0:0:1:0): dev_openings 1024 > (pass1:mrsas0:0:1:0): dev_active 0 > (pass1:mrsas0:0:1:0): devq_openings 1024 > (pass1:mrsas0:0:1:0): devq_queued 0 > (pass1:mrsas0:0:1:0): held 0 > (pass1:mrsas0:0:1:0): mintags 2 > (pass1:mrsas0:0:1:0): maxtags 1024 >=20 > Value 1024 is hard coded for my testing. In MegaRaid controller and SAS- > HBA Driver read max commands value from FW. > Similar to "RequestCredit".. Different FW has different value, but they > are every time above 255. >=20 >=20 > When I run IOs dev_active stays in range of 0-255 only. See below > output when I run IOs on /dev/da1 and /dev/da13. I expect total > dev_openings should go beyond 255, which is not happening. >=20 >=20 > dhcp-135-24-192-127# camcontrol tags da1 -v > (pass1:mrsas0:0:1:0): dev_openings 832 > (pass1:mrsas0:0:1:0): dev_active 192 > (pass1:mrsas0:0:1:0): devq_openings 832 > (pass1:mrsas0:0:1:0): devq_queued 0 > (pass1:mrsas0:0:1:0): held 0 > (pass1:mrsas0:0:1:0): mintags 2 > (pass1:mrsas0:0:1:0): maxtags 1024 > dhcp-135-24-192-127# camcontrol tags da13 -v > (pass13:mrsas0:0:13:0): dev_openings 881 > (pass13:mrsas0:0:13:0): dev_active 143 > (pass13:mrsas0:0:13:0): devq_openings 881 > (pass13:mrsas0:0:13:0): devq_queued 0 > (pass13:mrsas0:0:13:0): held 0 > (pass13:mrsas0:0:13:0): mintags 2 > (pass13:mrsas0:0:13:0): maxtags 1024 >=20 >=20 >=20 >=20 > Jim: > Below is my API call. I have hard code value "queue_depth" =3D 1024 >=20 > sc->sim_0 =3D cam_sim_alloc(mrsas_action, mrsas_poll, "mrsas", sc, > device_get_unit(sc->mrsas_dev), &sc->sim_lock, queue_depth, > queue_depth, devq); >=20 > ~ Kashyap >=20 > > > > Ken > > -- > > Kenneth Merry > > ken@FreeBSD.ORG > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 24 09:46:26 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D5DEC5E9; Thu, 24 Jan 2013 09:46:26 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop04.sare.net (proxypop04.sare.net [194.30.0.65]) by mx1.freebsd.org (Postfix) with ESMTP id 5F0EA83F; Thu, 24 Jan 2013 09:46:26 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 0714F9DEB8D; Thu, 24 Jan 2013 10:46:13 +0100 (CET) From: Borja Marcos Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Problem adding SCSI quirks for a SSD, 4K sector and ZFS Date: Thu, 24 Jan 2013 10:46:23 +0100 Message-Id: <492280E6-E3EE-4540-92CE-C535C8943CCF@sarenet.es> To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1085) X-Mailer: Apple Mail (2.1085) Cc: FreeBSD Filesystems X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jan 2013 09:46:26 -0000 Hello, Crossposting to FreeBSD-fs, as I am wondering if I have had a problem = with ZFS and sector size detection as well. I am doing tests with an OCZ Vertex 4 connected to a SAS backplane. < OCZ-VERTEX4 1.5> at scbus6 target 22 lun 0 = (pass19,da15) (The blank before "OCZ" really appears there) pass19: < OCZ-VERTEX4 1.5> Fixed Direct Access SCSI-5 device=20 pass19: Serial Number OCZ-1SVG6KZ2YRMSS8E1 pass19: 3.300MB/s transfers I am bypassing an "aac" RAID card so that the disks are directly = attached to the da driver, instead of relying on the so-called JBOD = feature. I have had a weird problem, with the disk being unresponsive to the = REQUEST CAPACITY(16) command. Weird, seems it timeouts. So, just to complete the tests, I have added a quirk to scsi_da.c. = Anyway, I also need the disk to be recognized as a 4K sector drive. I created a new quirk, called it DA_Q_NO_RC16, and added an entry to = the quirk table, so that these drives are recognized as 4K drives and = the driver doesn't try to send a RC(16) command. diff scsi_da.c.orig scsi_da.c 93c93,94 < DA_Q_4K =3D 0x08 --- > DA_Q_4K =3D 0x08, > DA_Q_NO_RC16 =3D 0x10 811a813,817 > /* OCZ Vertex 4 firmware 1.5 */ > { T_DIRECT, SIP_MEDIA_FIXED, "", "OCZ-VERTEX4", "*" }, > /*quirks*/DA_Q_NO_RC16 | DA_Q_4K > }, > { 1635,1636c1641,1646 < /* Predict whether device may support READ CAPACITY(16). */ < if (SID_ANSI_REV(&cgd->inq_data) >=3D SCSI_REV_SPC3) { --- > /*=20 > * Predict whether device may support READ CAPACITY(16). > * BUT Some disks don't support RC(16) even though they should. > */ > if ((SID_ANSI_REV(&cgd->inq_data) >=3D SCSI_REV_SPC3)=20 > && !(softc->quirks & DA_Q_NO_RC16) ) { I think it's working. I haven't seen any more RC(16) errors, and the = disk is working fine. Anyway I am not sure I've done it right. After = adding the 4K quirk and rebooting, GEOM_PART complained that the = partitions weren't aligned to 4K /var/log/messages.0:Jan 23 16:01:30 kernel: GEOM_PART: partition 1 is = not aligned on 4096 bytes /var/log/messages.0:Jan 23 16:01:30 kernel: GEOM_PART: partition 2 is = not aligned on 4096 bytes So it seems it works. However, when using the disk for ZFS, it still = detects a 512 byte sector size, which is odd.=20 Jan 23 16:01:30 rasputin kernel: GEOM: new disk da15 Jan 23 16:01:30 rasputin kernel: da15 at aacp0 bus 0 scbus6 target 22 = lun 0 Jan 23 16:01:30 rasputin kernel: da15: < OCZ-VERTEX4 1.5> Fixed Direct = Access SCSI-5 device=20 Jan 23 16:01:30 rasputin kernel: da15: Serial Number = OCZ-1SVG6KZ2YRMSS8E1 Jan 23 16:01:30 rasputin kernel: da15: 3.300MB/s transfers Jan 23 16:01:30 rasputin kernel: da15: 488386MB (1000215216 512 byte = sectors: 255H 63S/T 62260C) diskinfo is returning a sector size of 512 bytes, and a stripesize of = 4096. Is this correct? ZFS is still detecting it as a 512 byte sector = disk. /dev/da15 512 # sectorsize 512110190592 # mediasize in bytes (477G) 1000215216 # mediasize in sectors 4096 # stripesize 0 # stripeoffset 62260 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. OCZ-1SVG6KZ2YRMSS8E1 # Disk ident. So, to summarize: If the quirk was working, should diskinfo return a sector size of 512 = bytes, or is it correct to show a "stripesize" of 4096? Do we have a bug either on ZFS or the disk drivers? The same experiment = on another system (both are 9.1-RELEASE) and a similar drive attached to = a SATA controller, also adding a 4K sector quirk for it, defines a = stripe size instead of a sector size. Thanks, Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 24 11:19:24 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4D869D1D; Thu, 24 Jan 2013 11:19:24 +0000 (UTC) (envelope-from prvs=1736dd70aa=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 5F530F73; Thu, 24 Jan 2013 11:19:23 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50001833957.msg; Thu, 24 Jan 2013 11:19:20 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 24 Jan 2013 11:19:20 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1736dd70aa=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Borja Marcos" , References: <492280E6-E3EE-4540-92CE-C535C8943CCF@sarenet.es> Subject: Re: Problem adding SCSI quirks for a SSD, 4K sector and ZFS Date: Thu, 24 Jan 2013 11:19:52 -0000 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0014_01CDFA24.BAC45C90" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD Filesystems X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jan 2013 11:19:24 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_0014_01CDFA24.BAC45C90 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit ----- Original Message ----- From: "Borja Marcos" To: Cc: "FreeBSD Filesystems" Sent: Thursday, January 24, 2013 9:46 AM Subject: Problem adding SCSI quirks for a SSD, 4K sector and ZFS > > Hello, > > Crossposting to FreeBSD-fs, as I am wondering if I have had a problem with ZFS and sector size detection as well. > > I am doing tests with an OCZ Vertex 4 connected to a SAS backplane. > > < OCZ-VERTEX4 1.5> at scbus6 target 22 lun 0 (pass19,da15) > > (The blank before "OCZ" really appears there) > > pass19: < OCZ-VERTEX4 1.5> Fixed Direct Access SCSI-5 device > pass19: Serial Number OCZ-1SVG6KZ2YRMSS8E1 > pass19: 3.300MB/s transfers > > I am bypassing an "aac" RAID card so that the disks are directly attached to the da driver, instead of relying on the so-called > JBOD feature. > > I have had a weird problem, with the disk being unresponsive to the REQUEST CAPACITY(16) command. Weird, seems it timeouts. > > So, just to complete the tests, I have added a quirk to scsi_da.c. Anyway, I also need the disk to be recognized as a 4K sector > drive. > > I created a new quirk, called it DA_Q_NO_RC16, and added an entry to the quirk table, so that these drives are recognized as 4K > drives and the driver doesn't try to send a RC(16) command. > > diff scsi_da.c.orig scsi_da.c > 93c93,94 > < DA_Q_4K = 0x08 > --- >> DA_Q_4K = 0x08, >> DA_Q_NO_RC16 = 0x10 > 811a813,817 >> /* OCZ Vertex 4 firmware 1.5 */ >> { T_DIRECT, SIP_MEDIA_FIXED, "", "OCZ-VERTEX4", "*" }, >> /*quirks*/DA_Q_NO_RC16 | DA_Q_4K >> }, >> { > 1635,1636c1641,1646 > < /* Predict whether device may support READ CAPACITY(16). */ > < if (SID_ANSI_REV(&cgd->inq_data) >= SCSI_REV_SPC3) { > --- >> /* >> * Predict whether device may support READ CAPACITY(16). >> * BUT Some disks don't support RC(16) even though they should. >> */ >> if ((SID_ANSI_REV(&cgd->inq_data) >= SCSI_REV_SPC3) >> && !(softc->quirks & DA_Q_NO_RC16) ) { > > > > I think it's working. I haven't seen any more RC(16) errors, and the disk is working fine. Anyway I am not sure I've done it > right. After adding the 4K quirk and rebooting, GEOM_PART complained that the partitions weren't aligned to 4K > > /var/log/messages.0:Jan 23 16:01:30 kernel: GEOM_PART: partition 1 is not aligned on 4096 bytes > /var/log/messages.0:Jan 23 16:01:30 kernel: GEOM_PART: partition 2 is not aligned on 4096 bytes > > So it seems it works. However, when using the disk for ZFS, it still detects a 512 byte sector size, which is odd. > > Jan 23 16:01:30 rasputin kernel: GEOM: new disk da15 > Jan 23 16:01:30 rasputin kernel: da15 at aacp0 bus 0 scbus6 target 22 lun 0 > Jan 23 16:01:30 rasputin kernel: da15: < OCZ-VERTEX4 1.5> Fixed Direct Access SCSI-5 device > Jan 23 16:01:30 rasputin kernel: da15: Serial Number OCZ-1SVG6KZ2YRMSS8E1 > Jan 23 16:01:30 rasputin kernel: da15: 3.300MB/s transfers > Jan 23 16:01:30 rasputin kernel: da15: 488386MB (1000215216 512 byte sectors: 255H 63S/T 62260C) > > > diskinfo is returning a sector size of 512 bytes, and a stripesize of 4096. Is this correct? ZFS is still detecting it as a 512 > byte sector disk. > > /dev/da15 > 512 # sectorsize > 512110190592 # mediasize in bytes (477G) > 1000215216 # mediasize in sectors > 4096 # stripesize > 0 # stripeoffset > 62260 # Cylinders according to firmware. > 255 # Heads according to firmware. > 63 # Sectors according to firmware. > OCZ-1SVG6KZ2YRMSS8E1 # Disk ident. > > > > So, to summarize: > > If the quirk was working, should diskinfo return a sector size of 512 bytes, or is it correct to show a "stripesize" of 4096? > > Do we have a bug either on ZFS or the disk drivers? The same experiment on another system (both are 9.1-RELEASE) and a similar > drive attached to a SATA controller, also adding a 4K sector quirk for it, defines a stripe size instead of a sector size. Simple answer is ZFS doesn't understand quirks. The attached patch does what you're looking for along with a few other things, see notes at the top for details. Its not a final version as there's still some discussion about implementation details but it should do what your looking for. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. ------=_NextPart_000_0014_01CDFA24.BAC45C90 Content-Type: application/octet-stream; name="zzz-zfs-ashift-fix.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="zzz-zfs-ashift-fix.patch" Changes zfs zpool initial / desired ashift to be based off stripesize=0A= instead of sectorsize making it compatible with drives marked with=0A= the 4k sector size quirk.=0A= =0A= Without the correct min block size BIO_DELETE requests passed to=0A= a large number of current SSD's via TRIM don't actually perform=0A= any LBA TRIM so its vital for the correct operation of TRIM to get=0A= the correct min block size.=0A= =0A= To do this we added the additional dashift (desired ashift) to=0A= vdev_open_func_t calls. This was needed as just updating ashift to=0A= be based off stripesize would mean that a devices reported minimum=0A= transfer size (ashift) could increase and that in turn would cause=0A= member devices to be unusable and hence break pools with error=0A= ZFS-8000-5E.=0A= =0A= The global minimum ashift used for new zpools can now also be=0A= tuned using the vfs.zfs.min_create_ashift sysctl. This defaults=0A= to 12 (4096 byte blocks) in order to optimise for newer disks which=0A= are migrating from 512 to 4096 byte sectors.=0A= =0A= The value of vfs.zfs.min_create_ashift is limited to min of=0A= SPA_MINBLOCKSHIFT (9) and a max of SPA_MAXBLOCKSHIFT (17).=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_disk.c.orig = 2011-06-06 09:36:46.000000000 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_disk.c = 2012-11-02 14:47:55.293668071 +0000=0A= @@ -32,6 +32,8 @@=0A= #include =0A= #include =0A= =0A= +extern int zfs_min_ashift;=0A= +=0A= /*=0A= * Virtual device vector for disks.=0A= */=0A= @@ -103,7 +105,7 @@=0A= }=0A= =0A= static int=0A= -vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift)=0A= +vdev_disk_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift, uint64_t = *dashift)=0A= {=0A= spa_t *spa =3D vd->vdev_spa;=0A= vdev_disk_t *dvd;=0A= @@ -284,7 +286,7 @@=0A= }=0A= =0A= /*=0A= - * Determine the device's minimum transfer size.=0A= + * Determine the device's minimum and desired transfer size.=0A= * If the ioctl isn't supported, assume DEV_BSIZE.=0A= */=0A= if (ldi_ioctl(dvd->vd_lh, DKIOCGMEDIAINFOEXT, (intptr_t)&dkmext,=0A= @@ -292,6 +294,7 @@=0A= dkmext.dki_pbsize =3D DEV_BSIZE;=0A= =0A= *ashift =3D highbit(MAX(dkmext.dki_pbsize, SPA_MINBLOCKSIZE)) - 1;=0A= + *dashift =3D highbit(MAX(dkmext.dki_pbsize, (1ULL << zfs_min_ashift))) = - 1;=0A= =0A= /*=0A= * Clear the nowritecache bit, so that on a vdev_reopen() we will=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_file.c.orig = 2012-01-05 22:31:25.000000000 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_file.c = 2012-11-02 14:47:38.252107541 +0000=0A= @@ -30,6 +30,8 @@=0A= #include =0A= #include =0A= =0A= +extern int zfs_min_ashift;=0A= +=0A= /*=0A= * Virtual device vector for files.=0A= */=0A= @@ -47,7 +49,7 @@=0A= }=0A= =0A= static int=0A= -vdev_file_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift)=0A= +vdev_file_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift, uint64_t = *dashift)=0A= {=0A= vdev_file_t *vf;=0A= vnode_t *vp;=0A= @@ -127,6 +129,7 @@=0A= =0A= *psize =3D vattr.va_size;=0A= *ashift =3D SPA_MINBLOCKSHIFT;=0A= + *dashift =3D zfs_min_ashift;=0A= =0A= return (0);=0A= }=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c.orig = 2012-11-02 12:20:15.918986181 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c = 2012-11-02 14:47:48.135273692 +0000=0A= @@ -36,6 +36,8 @@=0A= #include =0A= #include =0A= =0A= +extern int zfs_min_ashift;=0A= +=0A= /*=0A= * Virtual device vector for GEOM.=0A= */=0A= @@ -408,7 +410,7 @@=0A= }=0A= =0A= static int=0A= -vdev_geom_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift)=0A= +vdev_geom_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift, uint64_t = *dashift)=0A= {=0A= struct g_provider *pp;=0A= struct g_consumer *cp;=0A= @@ -494,9 +496,10 @@=0A= *psize =3D pp->mediasize;=0A= =0A= /*=0A= - * Determine the device's minimum transfer size.=0A= + * Determine the device's minimum and desired transfer size.=0A= */=0A= *ashift =3D highbit(MAX(pp->sectorsize, SPA_MINBLOCKSIZE)) - 1;=0A= + *dashift =3D highbit(MAX(pp->stripesize, (1ULL << zfs_min_ashift))) - = 1;=0A= =0A= /*=0A= * Clear the nowritecache settings, so that on a vdev_reopen()=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c.orig = 2012-07-03 11:49:22.342245151 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c = 2012-07-03 11:58:02.161948585 +0000=0A= @@ -127,7 +127,7 @@=0A= }=0A= =0A= static int=0A= -vdev_mirror_open(vdev_t *vd, uint64_t *asize, uint64_t *ashift)=0A= +vdev_mirror_open(vdev_t *vd, uint64_t *asize, uint64_t *ashift, = uint64_t *dashift)=0A= {=0A= int numerrors =3D 0;=0A= int lasterror =3D 0;=0A= @@ -150,6 +150,7 @@=0A= =0A= *asize =3D MIN(*asize - 1, cvd->vdev_asize - 1) + 1;=0A= *ashift =3D MAX(*ashift, cvd->vdev_ashift);=0A= + *dashift =3D MAX(*dashift, cvd->vdev_dashift);=0A= }=0A= =0A= if (numerrors =3D=3D vd->vdev_children) {=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_missing.c.orig = 2012-07-03 11:49:10.545275865 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_missing.c = 2012-07-03 11:58:07.670470640 +0000=0A= @@ -40,7 +40,7 @@=0A= =0A= /* ARGSUSED */=0A= static int=0A= -vdev_missing_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift)=0A= +vdev_missing_open(vdev_t *vd, uint64_t *psize, uint64_t *ashift, = uint64_t *dashift)=0A= {=0A= /*=0A= * Really this should just fail. But then the root vdev will be in the=0A= @@ -50,6 +50,7 @@=0A= */=0A= *psize =3D 0;=0A= *ashift =3D 0;=0A= + *dashift =3D 0;=0A= return (0);=0A= }=0A= =0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c.orig = 2012-07-03 11:49:03.675875505 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c = 2012-07-03 11:58:15.334806334 +0000=0A= @@ -1447,7 +1447,7 @@=0A= }=0A= =0A= static int=0A= -vdev_raidz_open(vdev_t *vd, uint64_t *asize, uint64_t *ashift)=0A= +vdev_raidz_open(vdev_t *vd, uint64_t *asize, uint64_t *ashift, uint64_t = *dashift)=0A= {=0A= vdev_t *cvd;=0A= uint64_t nparity =3D vd->vdev_nparity;=0A= @@ -1476,6 +1476,7 @@=0A= =0A= *asize =3D MIN(*asize - 1, cvd->vdev_asize - 1) + 1;=0A= *ashift =3D MAX(*ashift, cvd->vdev_ashift);=0A= + *dashift =3D MAX(*dashift, cvd->vdev_dashift);=0A= }=0A= =0A= *asize *=3D vd->vdev_children;=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_root.c.orig = 2012-07-03 11:49:27.901760380 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_root.c = 2012-07-03 11:58:19.704427068 +0000=0A= @@ -50,7 +50,7 @@=0A= }=0A= =0A= static int=0A= -vdev_root_open(vdev_t *vd, uint64_t *asize, uint64_t *ashift)=0A= +vdev_root_open(vdev_t *vd, uint64_t *asize, uint64_t *ashift, uint64_t = *dashift)=0A= {=0A= int lasterror =3D 0;=0A= int numerrors =3D 0;=0A= @@ -78,6 +78,7 @@=0A= =0A= *asize =3D 0;=0A= *ashift =3D 0;=0A= + *dashift =3D 0;=0A= =0A= return (0);=0A= }=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c.orig = 2012-10-22 20:41:50.234005351 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c 2012-10-22 = 20:42:16.355805894 +0000=0A= @@ -1125,6 +1125,7 @@=0A= uint64_t osize =3D 0;=0A= uint64_t asize, psize;=0A= uint64_t ashift =3D 0;=0A= + uint64_t dashift =3D 0;=0A= =0A= ASSERT(vd->vdev_open_thread =3D=3D curthread ||=0A= spa_config_held(spa, SCL_STATE_ALL, RW_WRITER) =3D=3D = SCL_STATE_ALL);=0A= @@ -1154,7 +1155,7 @@=0A= return (ENXIO);=0A= }=0A= =0A= - error =3D vd->vdev_ops->vdev_op_open(vd, &osize, &ashift);=0A= + error =3D vd->vdev_ops->vdev_op_open(vd, &osize, &ashift, &dashift);=0A= =0A= /*=0A= * Reset the vdev_reopening flag so that we actually close=0A= @@ -1255,14 +1256,16 @@=0A= */=0A= vd->vdev_asize =3D asize;=0A= vd->vdev_ashift =3D MAX(ashift, vd->vdev_ashift);=0A= + vd->vdev_dashift =3D MAX(dashift, vd->vdev_dashift);=0A= } else {=0A= /*=0A= * Make sure the alignment requirement hasn't increased.=0A= */=0A= if (ashift > vd->vdev_top->vdev_ashift) {=0A= + printf("ZFS ashift open failure of %s (%ld > %ld)\n", vd->vdev_path, = ashift, vd->vdev_top->vdev_ashift);=0A= vdev_set_state(vd, B_TRUE, VDEV_STATE_CANT_OPEN,=0A= VDEV_AUX_BAD_LABEL);=0A= return (EINVAL);=0A= }=0A= }=0A= =0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c.orig = 2012-11-05 15:27:52.092194343 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c = 2012-11-05 15:53:26.449021023 +0000=0A= @@ -145,9 +145,12 @@=0A= #include =0A= =0A= static boolean_t vdev_trim_on_init =3D B_TRUE;=0A= +static boolean_t vdev_dashift_enable =3D B_TRUE;=0A= SYSCTL_DECL(_vfs_zfs_vdev);=0A= SYSCTL_INT(_vfs_zfs_vdev, OID_AUTO, trim_on_init, CTLFLAG_RW,=0A= &vdev_trim_on_init, 0, "Enable/disable full vdev trim on = initialisation");=0A= +SYSCTL_INT(_vfs_zfs_vdev, OID_AUTO, optimal_ashift, CTLFLAG_RW,=0A= + &vdev_dashift_enable, 0, "Enable/disable optimal ashift usage on = initialisation");=0A= =0A= /*=0A= * Basic routines to read and write from a vdev label.=0A= @@ -282,6 +285,16 @@=0A= vd->vdev_ms_array) =3D=3D 0);=0A= VERIFY(nvlist_add_uint64(nv, ZPOOL_CONFIG_METASLAB_SHIFT,=0A= vd->vdev_ms_shift) =3D=3D 0);=0A= + /*=0A= + * We use the max of ashift and dashift (the desired/optimal=0A= + * ashift), which is typically the stripesize of a device, to=0A= + * ensure we get the best performance from underlying devices.=0A= + * =0A= + * Its done here as it should only ever have an effect on new=0A= + * zpool creation.=0A= + */=0A= + if (vdev_dashift_enable)=0A= + vd->vdev_ashift =3D MAX(vd->vdev_ashift, vd->vdev_dashift);=0A= VERIFY(nvlist_add_uint64(nv, ZPOOL_CONFIG_ASHIFT,=0A= vd->vdev_ashift) =3D=3D 0);=0A= VERIFY(nvlist_add_uint64(nv, ZPOOL_CONFIG_ASIZE,=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h.orig = 2012-10-22 20:40:08.361577293 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/vdev_impl.h = 2012-10-22 21:02:52.447781800 +0000=0A= @@ -55,7 +55,7 @@=0A= /*=0A= * Virtual device operations=0A= */=0A= -typedef int vdev_open_func_t(vdev_t *vd, uint64_t *size, uint64_t = *ashift);=0A= +typedef int vdev_open_func_t(vdev_t *vd, uint64_t *size, uint64_t = *ashift, uint64_t *dashift);=0A= typedef void vdev_close_func_t(vdev_t *vd);=0A= typedef uint64_t vdev_asize_func_t(vdev_t *vd, uint64_t psize);=0A= typedef int vdev_io_start_func_t(zio_t *zio);=0A= @@ -119,6 +119,7 @@=0A= uint64_t vdev_asize; /* allocatable device capacity */=0A= uint64_t vdev_min_asize; /* min acceptable asize */=0A= uint64_t vdev_ashift; /* block alignment shift */=0A= + uint64_t vdev_dashift; /* desired blk alignment shift */=0A= uint64_t vdev_state; /* see VDEV_STATE_* #defines */=0A= uint64_t vdev_prevstate; /* used when reopening a vdev */=0A= vdev_ops_t *vdev_ops; /* vdev operations */=0A= --- sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c.orig = 2012-11-02 14:56:29.474248887 +0000=0A= +++ sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c 2012-11-03 = 01:27:28.066912403 +0000=0A= @@ -41,6 +41,30 @@=0A= #include =0A= #include =0A= =0A= +#define ZFS_MIN_ASHIFT SPA_MINBLOCKSHIFT=0A= +/*=0A= + * Max ashift - limited by how labels are accessed by zio_read_phys = using offsets=0A= + * within vdev_label_t=0A= + *=0A= + * If label access is fixed to work with ashift properly then the max = should be=0A= + * set to SPA_MAXBLOCKSHIFT=0A= + */=0A= +#define ZFS_MAX_ASHIFT 13=0A= +/*=0A= + * Optimum ashift - defaults to 12 which results in a min block size of = 4096 as=0A= + * this is the optimum value for newer disks which are migrating from = 512 to 4096=0A= + * byte sectors=0A= + */=0A= +#define ZFS_OPTIMUM_ASHIFT 12 =0A= +=0A= +/*=0A= + * Minimum ashift used when creating new pools=0A= + *=0A= + * This can be tuned using the sysctl vfs.zfs.min_create_ashift but is = limited=0A= + * to a min of ZFS_MIN_ASHIFT and a max of ZFS_MAX_ASHIFT=0A= + * =0A= + */=0A= +int zfs_min_ashift =3D MAX(SPA_MINBLOCKSHIFT, ZFS_OPTIMUM_ASHIFT);=0A= int zfs_no_write_throttle =3D 0;=0A= int zfs_write_limit_shift =3D 3; /* 1/8th of physical memory */=0A= int zfs_txg_synctime_ms =3D 1000; /* target millisecs to sync a txg */=0A= @@ -54,6 +78,9 @@=0A= =0A= static pgcnt_t old_physmem =3D 0;=0A= =0A= +#ifdef _KERNEL=0A= +static int min_ashift_sysctl(SYSCTL_HANDLER_ARGS);=0A= +=0A= SYSCTL_DECL(_vfs_zfs);=0A= TUNABLE_INT("vfs.zfs.no_write_throttle", &zfs_no_write_throttle);=0A= SYSCTL_INT(_vfs_zfs, OID_AUTO, no_write_throttle, CTLFLAG_RDTUN,=0A= @@ -78,6 +105,32 @@=0A= TUNABLE_QUAD("vfs.zfs.write_limit_override", &zfs_write_limit_override);=0A= SYSCTL_QUAD(_vfs_zfs, OID_AUTO, write_limit_override, CTLFLAG_RDTUN,=0A= &zfs_write_limit_override, 0, "");=0A= +SYSCTL_PROC(_vfs_zfs, OID_AUTO, min_create_ashift, CTLTYPE_INT | = CTLFLAG_RW,=0A= + &zfs_min_ashift, 0, min_ashift_sysctl, "I",=0A= + "Minimum ashift used when creating new pools");=0A= +=0A= +static int=0A= +min_ashift_sysctl(SYSCTL_HANDLER_ARGS)=0A= +{=0A= + int error, value;=0A= +=0A= + value =3D *(int *)arg1;=0A= +=0A= + error =3D sysctl_handle_int(oidp, &value, 0, req);=0A= +=0A= + if ((error !=3D 0) || (req->newptr =3D=3D NULL))=0A= + return (error);=0A= +=0A= + if (value < ZFS_MIN_ASHIFT)=0A= + value =3D ZFS_MIN_ASHIFT;=0A= + else if (value > ZFS_MAX_ASHIFT)=0A= + value =3D ZFS_MAX_ASHIFT;=0A= +=0A= + *(int *)arg1 =3D value;=0A= +=0A= + return (0);=0A= +}=0A= +#endif=0A= =0A= int=0A= dsl_pool_open_special_dir(dsl_pool_t *dp, const char *name, dsl_dir_t = **ddp)=0A= ------=_NextPart_000_0014_01CDFA24.BAC45C90-- From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 24 16:10:01 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3FE96B29; Thu, 24 Jan 2013 16:10:01 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop04.sare.net (proxypop04.sare.net [194.30.0.65]) by mx1.freebsd.org (Postfix) with ESMTP id 03B69230; Thu, 24 Jan 2013 16:10:00 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop04.sare.net (Postfix) with ESMTPSA id 552499DD631; Thu, 24 Jan 2013 17:09:46 +0100 (CET) Subject: Re: Problem adding SCSI quirks for a SSD, 4K sector and ZFS Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Borja Marcos X-Priority: 3 In-Reply-To: Date: Thu, 24 Jan 2013 17:09:56 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <492280E6-E3EE-4540-92CE-C535C8943CCF@sarenet.es> To: Steven Hartland X-Mailer: Apple Mail (2.1085) Cc: FreeBSD Filesystems , freebsd-scsi@freebsd.org X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jan 2013 16:10:01 -0000 On Jan 24, 2013, at 12:19 PM, Steven Hartland wrote: >=20 > ----- Original Message ----- From: "Borja Marcos" > To: > Cc: "FreeBSD Filesystems" > Sent: Thursday, January 24, 2013 9:46 AM > Subject: Problem adding SCSI quirks for a SSD, 4K sector and ZFS >=20 >=20 >>=20 >> Hello, >>=20 >> Crossposting to FreeBSD-fs, as I am wondering if I have had a problem = with ZFS and sector size detection as well. >>=20 >> I am doing tests with an OCZ Vertex 4 connected to a SAS backplane. >>=20 >> < OCZ-VERTEX4 1.5> at scbus6 target 22 lun 0 = (pass19,da15) >>=20 >> (The blank before "OCZ" really appears there) >>=20 >> pass19: < OCZ-VERTEX4 1.5> Fixed Direct Access SCSI-5 device >> pass19: Serial Number OCZ-1SVG6KZ2YRMSS8E1 >> pass19: 3.300MB/s transfers >>=20 >> I am bypassing an "aac" RAID card so that the disks are directly = attached to the da driver, instead of relying on the so-called JBOD = feature. >>=20 >> I have had a weird problem, with the disk being unresponsive to the = REQUEST CAPACITY(16) command. Weird, seems it timeouts. >>=20 >> So, just to complete the tests, I have added a quirk to scsi_da.c. = Anyway, I also need the disk to be recognized as a 4K sector drive. >>=20 (...) > Simple answer is ZFS doesn't understand quirks. The attached patch = does > what you're looking for along with a few other things, see notes at > the top for details. >=20 > Its not a final version as there's still some discussion about > implementation details but it should do what your looking for. Oh, sorrry for the confusion! I assumed that the da driver would = announce a 4 KB sector instead of the 512 bytes due to the quirks.=20 Which brings an interesting question, I might have a potential time = bomb in my experimental machine at home. It's a ZFS mirror with two 500 = MB disks. One of the has 512 byte sectors, the other is "advanced = format".=20 Borja. From owner-freebsd-scsi@FreeBSD.ORG Thu Jan 24 21:08:05 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 08D59D9C; Thu, 24 Jan 2013 21:08:05 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 435F2723; Thu, 24 Jan 2013 21:08:03 +0000 (UTC) Received: from digsys200-136.pip.digsys.bg (digsys200-136.pip.digsys.bg [193.68.136.200]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.5/8.14.5) with ESMTP id r0OKkb3B079889 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 24 Jan 2013 22:46:38 +0200 (EET) (envelope-from daniel@digsys.bg) Subject: Re: Problem adding SCSI quirks for a SSD, 4K sector and ZFS Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Content-Type: text/plain; charset=us-ascii From: Daniel Kalchev X-Priority: 3 In-Reply-To: Date: Thu, 24 Jan 2013 22:46:37 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <92BD25C7-FC6C-41AF-80AE-05AAD4CD4659@digsys.bg> References: <492280E6-E3EE-4540-92CE-C535C8943CCF@sarenet.es> To: Borja Marcos X-Mailer: Apple Mail (2.1499) Cc: FreeBSD Filesystems , freebsd-scsi@freebsd.org, Steven Hartland X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jan 2013 21:08:05 -0000 On Jan 24, 2013, at 6:09 PM, Borja Marcos wrote: > Oh, sorrry for the confusion! I assumed that the da driver would = announce a 4 KB sector instead of the 512 bytes due to the quirks.=20 >=20 > Which brings an interesting question, I might have a potential time = bomb in my experimental machine at home. It's a ZFS mirror with two 500 = MB disks. One of the has 512 byte sectors, the other is "advanced = format".=20 If you did not create your zpool with ashift=3D12, then you have the = bomb already. Disks with different native sector size coexist quite = happily in the same zpool. If the storage does not lie, ZFS will in fact = use the largest 'sector size' and everything will be ok. Many (most) = advanced format drives however lie about their sector size and pretend = to have 512 byte sectors. If you create an zpool with such a drive and = an 'normal' 512 byte drive without explicitly requesting 4k alignment, = thing are bad. Daniel=