From owner-freebsd-scsi@FreeBSD.ORG Thu Aug 30 13:40:44 2007 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78F4916A420 for ; Thu, 30 Aug 2007 13:40:44 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id 5605313C47E for ; Thu, 30 Aug 2007 13:40:44 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.local (209-163-168-124.static.twtelecom.net [209.163.168.124]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id l7UDeINA013907; Thu, 30 Aug 2007 08:40:21 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <46D6C8C2.3000504@freebsd.org> Date: Thu, 30 Aug 2007 08:40:18 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Scott Long References: <71d0ebb0708291245g79d2141fx73cc8a6e76875944@mail.gmail.com> <46D5E17F.3070403@samsco.org> <71d0ebb0708291416v17351c65u7ccc1b7bbe0271d2@mail.gmail.com> <46D5E5B1.207@samsco.org> <71d0ebb0708291506i49649a60l8006deafb20891ac@mail.gmail.com> <46D63710.1020103@freebsd.org> <46D64020.3000503@samsco.org> In-Reply-To: <46D64020.3000503@samsco.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.7 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: FREEBSD - SCSI - LIST Subject: Re: performance with LSI SAS 1064 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2007 13:40:44 -0000 Scott Long wrote: > Eric Anderson wrote: >> Lutieri G. wrote: >>> I've make a test with dd command: >>> >>> # time dd if=/dev/zero of=./8gbfile bs=1024k count=8192 >>> 8192+0 records in >>> 8192+0 records out >>> 8589934592 bytes transferred in 155.653213 secs (55186362 bytes/sec) >>> 0.007u 25.129s 2:35.69 16.1% 55+6039k 117+68628io 0pf+0w >>> >>> in other terminal i ran iostat while dd were running and I get this: >>> >>> # iostat -I 1 >>> tty da0 pass0 cpu >>> tin tout KB/t xfrs MB KB/t xfrs MB us ni sy in id >>> 1 61 25.23 52958 1305.07 0.00 0 0.00 0 0 8 0 91 >>> 0 184 127.48 434 54.03 0.00 0 0.00 0 0 5 0 95 >>> 0 61 127.49 440 54.78 0.00 0 0.00 0 0 5 0 95 >>> 0 61 127.75 445 55.52 0.00 0 0.00 0 0 5 0 95 >>> 0 61 127.49 442 55.03 0.00 0 0.00 0 0 4 0 96 >>> 0 61 127.49 436 54.28 0.00 0 0.00 0 0 5 0 95 >>> 0 61 125.27 425 51.99 0.00 0 0.00 0 0 5 1 94 >>> 0 61 118.14 393 45.34 0.00 0 0.00 0 0 3 0 97 >>> >>> average 54MB/s with or without hw.mpt.enable_sata_wc seted in >>> loader.conf file. >>> >>> is it a normal speed for this adapter?! >>> >>> >> >> >> I'm confused - you said in your first post you were getting 3MB/s, >> where above you show something like 55MB/s. >> >> You didn't say what kind of disks, or how many, the configuration, etc >> - so it's hard to answer much. The 55MB/s seems pretty decent for >> many hard drives in a sequential use state (which is what dd tests >> really). >> >> Your errors before were probably caused because your queue depth is >> set to 255 (or 256?) and the adapter can't do that many. You should >> use camcontrol to reduce it, to maybe 32. See the camcontrol man page >> for the right usage. It's something that needs setting on every boot, >> so a startup file is a good place for it maybe. >> >> Eric >> >> > > Well, if he's using SATA (which I kinda assumed originally without > asking) then queue depth isn't going to matter; the MPT driver has no > interaction with how SATA NCQ operates, if he even has a rev of the > LSI chip that supports NCQ at all. If he's using SAS, then queue > depth will only be a minor factor, CAM is pretty good at autosizing > the depth with minimal impact. Now, if he's using SAS disks then > the boot tunable that I gave him will indeed have no impact at all. > > I believe that the Sun 4100 uses 2.5" disks, whether SATA or SAS. > 54MB/s is not all that bad for disks of this size. It's pretty close > to what I would expect, actually. If he's using a SAS Seagate 15k rpm 2.5" drive, he could see much better than 55MB/s. I have an LSI (PCI-X 133, model 1064 as he does) with some Seagate 15k RPM drives, and I can get 100MB/s. Tests were done on FreeBSD 7-CURRENT, with write caching enabled on the drive. Here's some of the numbers: 1k block read size: 6390033 bytes/sec 2k block read size: 12257594 bytes/sec 4k block read size: 23071775 bytes/sec 8k block read size: 41858586 bytes/sec 16k block read size: 71242623 bytes/sec 32k block read size: 108019212 bytes/sec 64k block read size: 108043811 bytes/sec 128k block read size: 108881295 bytes/sec 256k block read size: 108509827 bytes/sec 512k block read size: 108670186 bytes/sec 1024k block read size: 108553724 bytes/sec 1k block write size: 6199645 bytes/sec 2k block write size: 12169083 bytes/sec 4k block write size: 23246388 bytes/sec 8k block write size: 42133751 bytes/sec 16k block write size: 70968273 bytes/sec 32k block write size: 108306742 bytes/sec 64k block write size: 108142985 bytes/sec 128k block write size: 109156720 bytes/sec 256k block write size: 109164252 bytes/sec 512k block write size: 109795665 bytes/sec 1024k block write size: 109202660 bytes/sec Doing latency tests on /dev/da0 /dev/da0 512 # sectorsize 73407820800 # mediasize in bytes (68G) 143374650 # mediasize in sectors 8924 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. Seek times: Full stroke: 250 iter in 1.943102 sec = 7.772 msec Half stroke: 250 iter in 1.567395 sec = 6.270 msec Quarter stroke: 500 iter in 1.671074 sec = 3.342 msec Short forward: 400 iter in 1.159763 sec = 2.899 msec Short backward: 400 iter in 1.014000 sec = 2.535 msec Seq outer: 2048 iter in 0.322013 sec = 0.157 msec Seq inner: 2048 iter in 0.493810 sec = 0.241 msec Transfer rates: outside: 102400 kbytes in 0.968299 sec = 105752 kbytes/sec middle: 102400 kbytes in 1.088158 sec = 94104 kbytes/sec inside: 102400 kbytes in 1.361127 sec = 75232 kbytes/sec Doing read tests on /dev/da0 READ: 1k block size: 102400+0 records in 102400+0 records out 104857600 bytes transferred in 16.409555 secs (6390033 bytes/sec) READ: 2k block size: 51200+0 records in 51200+0 records out 104857600 bytes transferred in 8.554501 secs (12257594 bytes/sec) READ: 4k block size: 25600+0 records in 25600+0 records out 104857600 bytes transferred in 4.544843 secs (23071775 bytes/sec) READ: 8k block size: 12800+0 records in 12800+0 records out 104857600 bytes transferred in 2.505044 secs (41858586 bytes/sec) READ: 16k block size: 6400+0 records in 6400+0 records out 104857600 bytes transferred in 1.471838 secs (71242623 bytes/sec) READ: 32k block size: 3200+0 records in 3200+0 records out 104857600 bytes transferred in 0.970731 secs (108019212 bytes/sec) READ: 64k block size: 1600+0 records in 1600+0 records out 104857600 bytes transferred in 0.970510 secs (108043811 bytes/sec) READ: 128k block size: 800+0 records in 800+0 records out 104857600 bytes transferred in 0.963045 secs (108881295 bytes/sec) READ: 256k block size: 400+0 records in 400+0 records out 104857600 bytes transferred in 0.966342 secs (108509827 bytes/sec) READ: 512k block size: 200+0 records in 200+0 records out 104857600 bytes transferred in 0.964916 secs (108670186 bytes/sec) READ: 1024k block size: 100+0 records in 100+0 records out 104857600 bytes transferred in 0.965951 secs (108553724 bytes/sec) Doing write tests on /dev/da0 WRITE: 1k block size: 102400+0 records in 102400+0 records out 104857600 bytes transferred in 16.913484 secs (6199645 bytes/sec) WRITE: 2k block size: 51200+0 records in 51200+0 records out 104857600 bytes transferred in 8.616722 secs (12169083 bytes/sec) WRITE: 4k block size: 25600+0 records in 25600+0 records out 104857600 bytes transferred in 4.510705 secs (23246388 bytes/sec) WRITE: 8k block size: 12800+0 records in 12800+0 records out 104857600 bytes transferred in 2.488684 secs (42133751 bytes/sec) WRITE: 16k block size: 6400+0 records in 6400+0 records out 104857600 bytes transferred in 1.477528 secs (70968273 bytes/sec) WRITE: 32k block size: 3200+0 records in 3200+0 records out 104857600 bytes transferred in 0.968154 secs (108306742 bytes/sec) WRITE: 64k block size: 1600+0 records in 1600+0 records out 104857600 bytes transferred in 0.969620 secs (108142985 bytes/sec) WRITE: 128k block size: 800+0 records in 800+0 records out 104857600 bytes transferred in 0.960615 secs (109156720 bytes/sec) WRITE: 256k block size: 400+0 records in 400+0 records out 104857600 bytes transferred in 0.960549 secs (109164252 bytes/sec) WRITE: 512k block size: 200+0 records in 200+0 records out 104857600 bytes transferred in 0.955025 secs (109795665 bytes/sec) WRITE: 1024k block size: 100+0 records in 100+0 records out 104857600 bytes transferred in 0.960211 secs (109202660 bytes/sec) Tests complete. Eric