From owner-freebsd-scsi@FreeBSD.ORG Mon Sep 22 15:47:01 2014 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5553AB9C for ; Mon, 22 Sep 2014 15:47:01 +0000 (UTC) Received: from cu01176a.smtpx.saremail.com (cu01176a.smtpx.saremail.com [195.16.150.151]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AD308BD1 for ; Mon, 22 Sep 2014 15:47:00 +0000 (UTC) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop03.sare.net (Postfix) with ESMTPSA id 1AC1B9DC60D for ; Mon, 22 Sep 2014 17:40:00 +0200 (CEST) From: Borja Marcos Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Subject: mpr vs mps performance Date: Mon, 22 Sep 2014 17:39:58 +0200 Message-Id: To: FreeBSD-scsi Mime-Version: 1.0 (Apple Message framework v1283) X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 15:47:01 -0000 Hello, I have been playing with the new SAS3 cards supported by the mpr driver, = and I=B4ve found out that they are, in the same hardware configuration, considerably slower writing data. Moreover, running two = simultaneous "bonnie" benchmarks (I am using SSDs, and one "bonnie" = sometimes hits 100% CPU usage, unable to really saturate the I/O) I see = the writing activity somewhat stalling, with disk bandwidth going from = 600 MB/s to around 50 for 20 seconds or so. I'd like to know if this matches anyone else's experiences. Also, I can = try and make some tests if needed. But for now it seems we will stick to = the SAS2 HBAs. The Bonnie results are: With mpr driver, SAS3: (each bonnie instance, so multiply the results by 2 to get the actual = bandwidth achieved) Seq output: (writing) Block: 292155 KB/s Rewrite: 139713 KB/s Seq input: Block: 862861 KB/s With mps driver: SAS2, again, total is 2x the following figures. Seq output: (writing) Block: 587950 KB/s Rewrite: 208239 KB/s Seq. input: (reading) Block: 842169 KB/s The storage is a ZFS pool with a 9-disk raidz2 vdev, made of Samsung = 840 EVO 1 TB SSDs. The pool has been created with an ashift of 12 (zpool = applied it thanks to the 4 KB block quirk for these SSDs) at scbus0 target 9 lun 0 (pass0,da0) at scbus0 target 10 lun 0 (pass1,da1) at scbus0 target 11 lun 0 (pass2,da2) at scbus0 target 12 lun 0 (pass3,da3) at scbus0 target 13 lun 0 (pass4,da4) at scbus0 target 14 lun 0 (pass5,da5) at scbus0 target 17 lun 0 (pass7,da6) at scbus0 target 18 lun 0 (pass8,da7) at scbus0 target 27 lun 0 = (pass14,da12) The mpr card details follow: Sep 17 09:49:39 elibm kernel: mpr0: port 0x3f00-0x3fff mem = 0x912f0000-0x912fffff irq 32 at device 0.0 on pci17 Sep 17 09:49:39 elibm kernel: mpr0: IOCFacts : Sep 17 09:49:39 elibm kernel: MsgVersion: 0x205 Sep 17 09:49:39 elibm kernel: HeaderVersion: 0x1d00 Sep 17 09:49:39 elibm kernel: IOCNumber: 0 Sep 17 09:49:39 elibm kernel: IOCExceptions: 0x0 Sep 17 09:49:39 elibm kernel: MaxChainDepth: 128 Sep 17 09:49:39 elibm kernel: NumberOfPorts: 1 Sep 17 09:49:39 elibm kernel: RequestCredit: 11264 Sep 17 09:49:39 elibm kernel: ProductID: 0x2221 Sep 17 09:49:39 elibm kernel: IOCRequestFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxInitiators: 1 Sep 17 09:49:39 elibm kernel: MaxTargets: 1024 Sep 17 09:49:39 elibm kernel: MaxSasExpanders: 14 Sep 17 09:49:39 elibm kernel: MaxEnclosures: 15 Sep 17 09:49:39 elibm kernel: HighPriorityCredit: 60 Sep 17 09:49:39 elibm kernel: MaxReplyDescriptorPostQueueDepth: 65504 Sep 17 09:49:39 elibm kernel: ReplyFrameSize: 32 Sep 17 09:49:39 elibm kernel: MaxVolumes: 0 Sep 17 09:49:39 elibm kernel: MaxDevHandle: 1047 Sep 17 09:49:39 elibm kernel: MaxPersistentEntries: 128 Sep 17 09:49:39 elibm kernel: mpr0: Firmware: 01.00.03.00, Driver: = 05.255.05.00-fbsd Sep 17 09:49:39 elibm kernel: mpr0: IOCCapabilities: = 3a85c And the mps card is a classic: Sep 22 17:18:24 elibm kernel: mps0: port 0x3f00-0x3fff mem = 0x90ebc000-0x90ebffff,0x912c0000-0x912fffff irq 32 at device 0.0 on = pci17 Sep 22 17:18:24 elibm kernel: mps0: Firmware: 18.00.00.00, Driver: = 19.00.00.00-fbsd Sep 22 17:18:24 elibm kernel: mps0: IOCCapabilities: = 1285c= The connected devices follow. Both use the same hardware (except for the = cables and HBA of course), but currently there's no way to check this = with the SAS3 card, as sas3ircu nor sas3flash detect it on FreeBSD. # sas2ircu 0 display LSI Corporation SAS2 IR Configuration Utility. Version 18.00.00.00 (2013.11.18)=20 Copyright (c) 2009-2013 LSI Corporation. All rights reserved.=20 Read configuration has been initiated for controller 0 ------------------------------------------------------------------------ Controller information ------------------------------------------------------------------------ Controller type : SAS2008 BIOS version : 7.35.00.00 Firmware version : 18.00.00.00 Channel description : 1 Serial Attached SCSI Initiator ID : 0 Maximum physical devices : 255 Concurrent commands supported : 3432 Slot : 3 Segment : 0 Bus : 17 Device : 0 Function : 0 RAID Support : No ------------------------------------------------------------------------ IR Volume information ------------------------------------------------------------------------ ------------------------------------------------------------------------ Physical device information ------------------------------------------------------------------------ Initiator at ID #0 Device is a Hard disk Enclosure # : 2 Slot # : 16 SAS Address : 5000c50-0-05b5-ce25 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE=20 Model Number : ST9146803SS =20 Firmware Revision : FS03 Serial No : 3SD02W5L GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 2 Slot # : 17 SAS Address : 5005076-0-3e8e-81a2 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08549F GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 18 SAS Address : 5005076-0-3e8e-81a3 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08548T GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 19 SAS Address : 5005076-0-3e8e-81a4 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08568E GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 20 SAS Address : 5005076-0-3e8e-81a5 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08547X GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 21 SAS Address : 5005076-0-3e8e-81a6 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08518Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 2 Slot # : 22 SAS Address : 5005076-0-3e8e-81a7 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08556K GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Enclosure services device Enclosure # : 2 Slot # : 255 SAS Address : 5005076-0-3e8e-81b9 State : Standby (SBY) Manufacturer : IBM-ESXS Model Number : SAS EXP BP =20 Firmware Revision : 61A6 Serial No : 00000006 GUID : N/A Protocol : SAS Device Type : Enclosure services device Device is a Hard disk Enclosure # : 3 Slot # : 0 SAS Address : 5005076-0-3e8e-86e9 State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08550R GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 1 SAS Address : 5005076-0-3e8e-86ea State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08911Y GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 2 SAS Address : 5005076-0-3e8e-86eb State : Ready (RDY) Size (in MB)/(in sectors) : 953869/1953525167 Manufacturer : ATA =20 Model Number : Samsung SSD 840=20 Firmware Revision : BB0Q Serial No : S1D9NEADA08811L GUID : N/A Protocol : SATA Drive Type : SATA_SSD Device is a Hard disk Enclosure # : 3 Slot # : 13 SAS Address : 5000c50-0-05b5-e531 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE=20 Model Number : ST9146803SS =20 Firmware Revision : FS03 Serial No : 3SD02STR GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 3 Slot # : 14 SAS Address : 5000c50-0-05b5-d489 State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE=20 Model Number : ST9146803SS =20 Firmware Revision : FS03 Serial No : 3SD02TV1 GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Hard disk Enclosure # : 3 Slot # : 15 SAS Address : 5000c50-0-05b5-f0ad State : Ready (RDY) Size (in MB)/(in sectors) : 140014/286749479 Manufacturer : SEAGATE=20 Model Number : ST9146803SS =20 Firmware Revision : FS03 Serial No : 3SD03F4C GUID : N/A Protocol : SAS Drive Type : SAS_HDD Device is a Enclosure services device Enclosure # : 3 Slot # : 255 SAS Address : 5005076-0-3e8e-86f9 State : Standby (SBY) Manufacturer : IBM-ESXS Model Number : SAS EXP BP =20 Firmware Revision : 61A6 Serial No : 00000006 GUID : N/A Protocol : SAS Device Type : Enclosure services device ------------------------------------------------------------------------ Enclosure information ------------------------------------------------------------------------ Enclosure# : 1 Logical ID : 500605b0:07ba2100 Numslots : 8 StartSlot : 0 Enclosure# : 2 Logical ID : 50050760:3e8e81a0 Numslots : 25 StartSlot : 0 Enclosure# : 3 Logical ID : 50050760:3e8e86e0 Numslots : 25 StartSlot : 0 ------------------------------------------------------------------------ SAS2IRCU: Command DISPLAY Completed Successfully. SAS2IRCU: Utility Completed Successfully.