Date: Wed, 12 Nov 2014 23:47:57 +0000 From: John <jwd@FreeBSD.org> To: FreeBSD-scsi <freebsd-scsi@freebsd.org> Subject: LSI 9300-8e performance experiences / pcie negotiation / Down reving protocol Message-ID: <20141112234757.GA85337@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
Hi Folks,
Does anyone have any experience with the LSI 9300 HBA adapter? I've
been putting together a new server with a pair of these cards going
into a quad-cabled SuperMicro storage enclosure and am not seeing the
performance I think I should.
I currently have 1 cable from each card going to each expander set. I
am unable to get better than 5.5GB/sec through-put.
pciconf shows the following for the cards:
mpr0@pci0:2:0:0: class=0x010700 card=0x30a01000 chip=0x00971000 rev=0x02 hdr=0x00
vendor = 'LSI Logic / Symbios Logic'
device = 'SAS3008 PCI-Express Fusion-MPT SAS-3'
class = mass storage
subclass = SAS
bar [10] = type I/O Port, range 32, base 0x5000, size 256, enabled
bar [14] = type Memory, range 64, base 0xc7500000, size 65536, enabled
cap 01[50] = powerspec 3 supports D0 D1 D2 D3 current D0
cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
speed 8.0(8.0)
cap 03[d0] = VPD
cap 05[a8] = MSI supports 1 message, 64 bit, vector masks
cap 11[c0] = MSI-X supports 96 messages, enabled
Table in map 0x14[0xe000], PBA in map 0x14[0xf000]
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0019[1e0] = PCIe Sec 1 lane errors 0
ecap 0004[1c0] = Power Budgeting 1
ecap 0016[190] = DPA 1
ecap 0010[150] = SRIOV 1
ecap 000e[148] = ARI 1
mpr1 is the same. I am curious about cap 10[68]. While the link widths
and speeds show correctly, it is a pcie3 card. All firmware is up-to-date
(I've actually tried all of them from 1 to 6 (no P3)). Cards are plugged
into PCI-Express 3 x16 slots. From dmidecode --type 9:
Designation: CPU1 SLOT3 PCI-E 3.0 X16
Type: x16 PCI Express 3 x16
3 x shelves: 24 drives ea. 72 total (da0 - da143)
# camcontrol inquiry ses1
pass49: <LSI SAS3x40 0501> Fixed Enclosure Services SCSI-5 device
pass49: Serial Number
pass49: 1200.000MB/s transfers, Command Queueing Enabled
# camcontrol inquiry da0
pass0: <SEAGATE ST1200MM0017 0002> Fixed Direct Access SCSI-6 device
pass0: Serial Number S3L0EENZ0000M446AG8K
pass0: 1200.000MB/s transfers, Command Queueing Enabled
gmultipath enabled
alternating i/o paths.
Booting the system with verbose enabled, I do see the following:
(probe0:mpr1:0:8:0): Down reving Protocol Version from 4 to 0?
It looks like 1 of these messages for every disk, coming from
cam/scsi/scsi_xpt.c:
if (cts->protocol_version > device->protocol_version) {
if (bootverbose) {
xpt_print(path, "Down reving Protocol "
"Version from %d to %d?\n", cts->protocol_version,
device->protocol_version);
}
cts->protocol_version = device->protocol_version;
}
If anyone has any thoughts please let me know.
Thanks,
John
Current test:
Write test files:
iozone -w -+N -+u -P 0 -i 0 -+n -s 16g -r 128k -t 24 -I -F /poollsi/01.tmp ...
export/import/rotate pool
Read test:
iozone -+N -+u -P 0 -i 1 -+n -s 16g -r 128k -t 24 -I -F /poollsi/01.tmp ...
Children see throughput for 24 readers = 5481194.48 KB/sec
Parent sees throughput for 24 readers = 5478389.11 KB/sec
Min throughput per process = 222687.09 KB/sec
Max throughput per process = 234463.22 KB/sec
Avg throughput per process = 228383.10 KB/sec
Min xfer = 15934592.00 KB
CPU utilization: Wall time 71.607 CPU time 563.762 CPU utilization 787.30 % (out of 24 cpus)
IOCFacts:
mpr1: <LSI SAS3008> port 0xe000-0xe0ff mem 0xfbd00000-0xfbd0ffff irq 56 at device 0.0 on pci135
mpr1: IOCFacts :
MsgVersion: 0x205
HeaderVersion: 0x2300
IOCNumber: 0
IOCExceptions: 0x0
MaxChainDepth: 128
NumberOfPorts: 1
RequestCredit: 10176
ProductID: 0x2221
IOCRequestFrameSize: 32
MaxInitiators: 32
MaxTargets: 1024
MaxSasExpanders: 64
MaxEnclosures: 65
HighPriorityCredit: 124
MaxReplyDescriptorPostQueueDepth: 65504
ReplyFrameSize: 32
MaxVolumes: 0
MaxDevHandle: 1128
MaxPersistentEntries: 128
mpr1: Firmware: 06.00.00.00, Driver: 05.255.05.00-fbsd
mpr1: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mpr1: attempting to allocate 1 MSI-X vectors (96 supported)
msi: routing MSI-X IRQ 324 to local APIC 0 vector 121
mpr1: using IRQ 324 for MSI-X
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141112234757.GA85337>