From owner-freebsd-performance@FreeBSD.ORG Wed Jul 7 00:02:22 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDB311065672 for ; Wed, 7 Jul 2010 00:02:22 +0000 (UTC) (envelope-from ssanders@softhammer.net) Received: from oproxy2-pub.bluehost.com (oproxy2-pub.bluehost.com [67.222.39.60]) by mx1.freebsd.org (Postfix) with SMTP id 78AAC8FC14 for ; Wed, 7 Jul 2010 00:02:22 +0000 (UTC) Received: (qmail 32176 invoked by uid 0); 6 Jul 2010 23:35:42 -0000 Received: from unknown (HELO host358.hostmonster.com) (66.147.240.158) by oproxy2.bluehost.com with SMTP; 6 Jul 2010 23:35:42 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=softhammer.net; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:X-Enigmail-Version:Content-Type:Content-Transfer-Encoding:X-Identified-User; b=CBTSshpQMZAW66LOykfUN98Tjt8pn+JueuO4tuLmW9eSTX0JmF4uKCzGHxA3v/sCwfPG6DgAXgErqSr1nm5tENx/CsfZELZNZpdb8+928lvV1hhj4ZHHECNI0DZuKckV; Received: from pool-74-96-233-244.washdc.fios.verizon.net ([74.96.233.244] helo=onyx.softhammer.net) by host358.hostmonster.com with esmtpa (Exim 4.69) (envelope-from ) id 1OWHfx-0003FF-N0 for freebsd-performance@freebsd.org; Tue, 06 Jul 2010 17:35:41 -0600 Message-ID: <4C33BDCC.1020004@softhammer.net> Date: Tue, 06 Jul 2010 19:35:40 -0400 From: Stephen Sanders User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.9) Gecko/20100430 Fedora/3.0.4-2.fc12 Lightning/1.0b2pre Thunderbird/3.0.4 MIME-Version: 1.0 To: freebsd-performance@freebsd.org X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Identified-User: {2492:host358.hostmonster.com:softhamm:softhammer.net} {sentby:smtp auth 74.96.233.244 authed with ssanders@softhammer.net} Subject: More Controllers != Higher Through Put X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2010 00:02:22 -0000 I'm wondering if anyone has heard of this. I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives and the motherboard servicing 2 7200 RPM Segate 1TB drives. The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives should deliver about 100MB/s. 1. The most the 4 disk array is developing is 250MB/s write performance while the 2 disk array is coming in at 90MB/s write performance. The 4 disk array seems slow. 2. Attempting to write to both arrays simultaneously causes the rate on the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s I'd expect the 4 disk array should look more like 300+MB/s while the 2 disk array is about right. I don't get why there should be a 'coupling' between the rates on separate controllers. The system is running FreeBSD 8.0, has 16GB of RAM in the system, and the test program is using O_DIRECT for writes in order to avoid the page daemon. Thanks From owner-freebsd-performance@FreeBSD.ORG Wed Jul 7 01:04:43 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 05EC5106564A for ; Wed, 7 Jul 2010 01:04:43 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from asmtpout023.mac.com (asmtpout023.mac.com [17.148.16.98]) by mx1.freebsd.org (Postfix) with ESMTP id E1C558FC08 for ; Wed, 7 Jul 2010 01:04:42 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; charset=us-ascii Received: from cswiger1.apple.com ([17.209.4.71]) by asmtp023.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec 16 2008; 32bit)) with ESMTPSA id <0L5500L7DXNU41A0@asmtp023.mac.com> for freebsd-performance@freebsd.org; Tue, 06 Jul 2010 18:04:42 -0700 (PDT) X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=6.0.2-1004200000 definitions=main-1007060132 X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.8161:2.4.5,1.2.40,4.0.166 definitions=2010-07-06_04:2010-02-06, 2010-07-07, 2010-07-06 signatures=0 From: Chuck Swiger In-reply-to: <4C33BDCC.1020004@softhammer.net> Date: Tue, 06 Jul 2010 18:04:42 -0700 Message-id: <70A92786-CB31-4A6E-9CC7-4FCEA3F83CE3@mac.com> References: <4C33BDCC.1020004@softhammer.net> To: Stephen Sanders X-Mailer: Apple Mail (2.1081) Cc: freebsd-performance@freebsd.org Subject: Re: More Controllers != Higher Through Put X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2010 01:04:43 -0000 Hi-- On Jul 6, 2010, at 4:35 PM, Stephen Sanders wrote: > I'm wondering if anyone has heard of this. > > I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives and the motherboard servicing 2 7200 RPM Segate 1TB drives. > > The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives should deliver about 100MB/s. > > 1. The most the 4 disk array is developing is 250MB/s write performance while the 2 disk array is coming in at 90MB/s write performance. The 4 disk array seems slow. > > 2. Attempting to write to both arrays simultaneously causes the rate on the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s > > I'd expect the 4 disk array should look more like 300+MB/s while the 2 disk array is about right. > > I don't get why there should be a 'coupling' between the rates on separate controllers. I'd imagine you are contending for memory bandwidth or similar bottleneck when you try to run both drive arrays at full tilt. As for the other question, if you want good write performance out of 4 drives, use RAID-10, not RAID-5 or -6. I regard it as normal for RAID-5 to perform slower than even a single drive when doing writes, and RAID-6 should be similar, although I haven't done anywhere near as much benchmarking with RAID-6 as with the other levels... Regards, -- -Chuck From owner-freebsd-performance@FreeBSD.ORG Wed Jul 7 14:12:47 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17157106564A for ; Wed, 7 Jul 2010 14:12:47 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 95A2C8FC14 for ; Wed, 7 Jul 2010 14:12:46 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1OWVMg-0007uz-Q5 for freebsd-performance@freebsd.org; Wed, 07 Jul 2010 16:12:42 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 07 Jul 2010 16:12:42 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 07 Jul 2010 16:12:42 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 07 Jul 2010 16:12:45 +0200 Lines: 73 Message-ID: References: <4C33BDCC.1020004@softhammer.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.9) Gecko/20100518 Thunderbird/3.0.4 In-Reply-To: <4C33BDCC.1020004@softhammer.net> X-Enigmail-Version: 1.0.1 Subject: Re: More Controllers != Higher Through Put X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2010 14:12:47 -0000 On 07/07/10 01:35, Stephen Sanders wrote: > I'm wondering if anyone has heard of this. > > I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives > and the motherboard servicing 2 7200 RPM Segate 1TB drives. So far so good. > The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives > should deliver about 100MB/s. Ok, so you've reduced the 4-drive array's write performance nearly to equivalent of 2 drives and the 2-drive array to 1 drive. It should be even worse for random IOs. Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID? For the 2-disk RAID-1 : you are probably using software RAID, right? (on-board "SATA RAID" controllers usually are just software implementations). > 1. The most the 4 disk array is developing is 250MB/s write performance This is too much. It almost looks like something is caching what shouldn't be cached. How did you get this result? I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with 100 MB/s drives. > while the 2 disk array is coming in at 90MB/s write performance. This is as expected - write performance of any size RAID-1 is equivalent of 1 drive or less. > The 4 disk array seems slow. Nope - the contrary should be true. It looks like you are doing something you shouldn't if you get that much performance, or your test is overly simplistic (e.g. you're testing cache). > 2. Attempting to write to both arrays simultaneously causes the rate on > the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s Are you running on an Atom CPU? What kind of system are you using? > I'd expect the 4 disk array should look more like 300+MB/s while the 2 > disk array is about right. No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction. Think about it: all the data needs to be a) written as-is to 2 of the drives, then b) parity/ECCs calculated and c) the same amount of data written to 2 more drives. You cannot get write performance of more than 2 drives equivalent in this scheme, and will probably be worse. For reads, only if your RAID controller is very, very smart (meaning: it probably isn't), you can recover some performance by using this parity/ECC data to reconstruct more data than is read from the two "plain" drives. I think ZFS does this in a limited way. > I don't get why there should be a 'coupling' between the rates on > separate controllers. This is the only thing which is puzzling a bit. I > The system is running FreeBSD 8.0, has 16GB of RAM in the system, and > the test program is using O_DIRECT for writes in order to avoid the page > daemon. You should use some benchmark which knows how to deal with OS cache, for example bonnie++ from the ports. Use a benchmark with random IO to see just how horrible your RAID-6 performance will be for random writes. From owner-freebsd-performance@FreeBSD.ORG Wed Jul 7 14:30:25 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D730C1065676 for ; Wed, 7 Jul 2010 14:30:25 +0000 (UTC) (envelope-from ssanders@softhammer.net) Received: from smtp-hq2.opnet.com (smtp-hq2.opnet.com [192.104.65.247]) by mx1.freebsd.org (Postfix) with ESMTP id 992698FC20 for ; Wed, 7 Jul 2010 14:30:25 +0000 (UTC) Received: from [172.16.12.251] (wtn12251.opnet.com [172.16.12.251]) by smtp.opnet.com (Postfix) with ESMTPSA id B7F29211009C; Wed, 7 Jul 2010 10:30:24 -0400 (EDT) Message-ID: <4C348F80.7040706@softhammer.net> Date: Wed, 07 Jul 2010 10:30:24 -0400 From: Stephen Sanders User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.10) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5 MIME-Version: 1.0 To: Ivan Voras References: <4C33BDCC.1020004@softhammer.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: More Controllers != Higher Through Put X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2010 14:30:26 -0000 Thanks for the response. Comments in inline --- On 7/7/2010 10:12 AM, Ivan Voras wrote: > On 07/07/10 01:35, Stephen Sanders wrote: > >> I'm wondering if anyone has heard of this. >> >> I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives >> and the motherboard servicing 2 7200 RPM Segate 1TB drives. >> > So far so good. > > >> The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives >> should deliver about 100MB/s. >> > Ok, so you've reduced the 4-drive array's write performance nearly to > equivalent of 2 drives and the 2-drive array to 1 drive. It should be > even worse for random IOs. > > Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID? > > For the 2-disk RAID-1 : you are probably using software RAID, right? > (on-board "SATA RAID" controllers usually are just software > implementations). > > The motherboard is an Intel 5000PSL with 2 - 3Ghz Xeon 5450 CPUs. The 3ware controller is a hardware raid implementation. The 2 disk RAID is probably done in software, its a BIOS configuration item. I should research that. An additional data point is that we've modified the kernel to use 128KB writes rather than the default of 64KB. >> 1. The most the 4 disk array is developing is 250MB/s write performance >> > This is too much. It almost looks like something is caching what > shouldn't be cached. How did you get this result? > > I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with > 100 MB/s drives. > > >> while the 2 disk array is coming in at 90MB/s write performance. >> > This is as expected - write performance of any size RAID-1 is equivalent > of 1 drive or less. > > >> The 4 disk array seems slow. >> > Nope - the contrary should be true. It looks like you are doing > something you shouldn't if you get that much performance, or your test > is overly simplistic (e.g. you're testing cache). > > The test program writes 32MB blocks to disk at a fixed rate with O_DIRECT set on the output file. I'll try this with bonnie and see what I get. >> 2. Attempting to write to both arrays simultaneously causes the rate on >> the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s >> > Are you running on an Atom CPU? What kind of system are you using? > > 2 Xeon 3Ghz E5450 >> I'd expect the 4 disk array should look more like 300+MB/s while the 2 >> disk array is about right. >> > No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction. > > Think about it: all the data needs to be a) written as-is to 2 of the > drives, then b) parity/ECCs calculated and c) the same amount of data > written to 2 more drives. You cannot get write performance of more than > 2 drives equivalent in this scheme, and will probably be worse. For > reads, only if your RAID controller is very, very smart (meaning: it > probably isn't), you can recover some performance by using this > parity/ECC data to reconstruct more data than is read from the two > "plain" drives. I think ZFS does this in a limited way. > > >> I don't get why there should be a 'coupling' between the rates on >> separate controllers. >> > This is the only thing which is puzzling a bit. I > > >> The system is running FreeBSD 8.0, has 16GB of RAM in the system, and >> the test program is using O_DIRECT for writes in order to avoid the page >> daemon. >> > You should use some benchmark which knows how to deal with OS cache, for > example bonnie++ from the ports. > > Use a benchmark with random IO to see just how horrible your RAID-6 > performance will be for random writes. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > From owner-freebsd-performance@FreeBSD.ORG Fri Jul 9 15:09:30 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5C28106566C for ; Fri, 9 Jul 2010 15:09:30 +0000 (UTC) (envelope-from jdixon@omniti.com) Received: from edge.omniti.com (smtp.omniti.com [8.8.38.6]) by mx1.freebsd.org (Postfix) with ESMTP id 7ADB78FC1C for ; Fri, 9 Jul 2010 15:09:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=omniti.com; s=s1024; c=relaxed/relaxed; q=dns/txt; i=@omniti.com; t=1278688164; h=From:Subject:Date:To; bh=4JESy4pcnlNxRWe0D3mXe8goYSw6mgxQuRTfP0+6sWM=; b=nrZqjH3qB+7EeV9eD/ODP9xjg4+raQbxVANIKHzuoj9KnsHd68vpG8EDACAaPFP0 vSc0kjl/j/Mj7iI5PEKAZAbD+19KJALzrQ2NpnwAyYM4hwqmyxpxpCMhw9jjJVWr g3jz3XfGl+BJTfR5U8HAyeomDzID6MTj2Zr7bkGKXpY=; Authentication-Results: edge smtp.user=jdixon@omniti.com; auth=pass (LOGIN) Received: from [68.55.0.29] ([68.55.0.29:51134] helo=omniti.com) by edge (envelope-from ) (ecelerity 2.2.2.35 r(26636M)) with ESMTPSA (cipher=AES256-SHA) id BD/2C-17327-4AB373C4; Fri, 09 Jul 2010 11:09:24 -0400 Date: Fri, 9 Jul 2010 11:09:20 -0400 From: Jason Dixon To: freebsd-performance@freebsd.org Message-ID: <20100709150920.GK4133@omniti.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) Subject: Last day to submit your Surge 2010 CFP! X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jul 2010 15:09:30 -0000 Today is your last chance to submit a CFP abstract for the 2010 Surge Scalability Conference. The event is taking place on Sept 30 and Oct 1, 2010 in Baltimore, MD. Surge focuses on case studies that address production failures and the re-engineering efforts that led to victory in Web Applications or Internet Architectures. You can find more information, including suggested topics and our current list of speakers, online: http://omniti.com/surge/2010 The final lineup should be available on the conference website next week. If you have questions about the CFP, attending Surge, or having your business sponsor/exhibit at Surge 2010, please contact us at surge@omniti.com. Thanks! -- Jason Dixon OmniTI Computer Consulting, Inc. jdixon@omniti.com 443.325.1357 x.241