Date: Wed, 07 Jul 2010 10:30:24 -0400 From: Stephen Sanders <ssanders@softhammer.net> To: Ivan Voras <ivoras@freebsd.org> Cc: freebsd-performance@freebsd.org Subject: Re: More Controllers != Higher Through Put Message-ID: <4C348F80.7040706@softhammer.net> In-Reply-To: <i1220f$4th$1@dough.gmane.org> References: <4C33BDCC.1020004@softhammer.net> <i1220f$4th$1@dough.gmane.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Thanks for the response. Comments in inline --- On 7/7/2010 10:12 AM, Ivan Voras wrote: > On 07/07/10 01:35, Stephen Sanders wrote: > >> I'm wondering if anyone has heard of this. >> >> I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives >> and the motherboard servicing 2 7200 RPM Segate 1TB drives. >> > So far so good. > > >> The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives >> should deliver about 100MB/s. >> > Ok, so you've reduced the 4-drive array's write performance nearly to > equivalent of 2 drives and the 2-drive array to 1 drive. It should be > even worse for random IOs. > > Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID? > > For the 2-disk RAID-1 : you are probably using software RAID, right? > (on-board "SATA RAID" controllers usually are just software > implementations). > > The motherboard is an Intel 5000PSL with 2 - 3Ghz Xeon 5450 CPUs. The 3ware controller is a hardware raid implementation. The 2 disk RAID is probably done in software, its a BIOS configuration item. I should research that. An additional data point is that we've modified the kernel to use 128KB writes rather than the default of 64KB. >> 1. The most the 4 disk array is developing is 250MB/s write performance >> > This is too much. It almost looks like something is caching what > shouldn't be cached. How did you get this result? > > I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with > 100 MB/s drives. > > >> while the 2 disk array is coming in at 90MB/s write performance. >> > This is as expected - write performance of any size RAID-1 is equivalent > of 1 drive or less. > > >> The 4 disk array seems slow. >> > Nope - the contrary should be true. It looks like you are doing > something you shouldn't if you get that much performance, or your test > is overly simplistic (e.g. you're testing cache). > > The test program writes 32MB blocks to disk at a fixed rate with O_DIRECT set on the output file. I'll try this with bonnie and see what I get. >> 2. Attempting to write to both arrays simultaneously causes the rate on >> the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s >> > Are you running on an Atom CPU? What kind of system are you using? > > 2 Xeon 3Ghz E5450 >> I'd expect the 4 disk array should look more like 300+MB/s while the 2 >> disk array is about right. >> > No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction. > > Think about it: all the data needs to be a) written as-is to 2 of the > drives, then b) parity/ECCs calculated and c) the same amount of data > written to 2 more drives. You cannot get write performance of more than > 2 drives equivalent in this scheme, and will probably be worse. For > reads, only if your RAID controller is very, very smart (meaning: it > probably isn't), you can recover some performance by using this > parity/ECC data to reconstruct more data than is read from the two > "plain" drives. I think ZFS does this in a limited way. > > >> I don't get why there should be a 'coupling' between the rates on >> separate controllers. >> > This is the only thing which is puzzling a bit. I > > >> The system is running FreeBSD 8.0, has 16GB of RAM in the system, and >> the test program is using O_DIRECT for writes in order to avoid the page >> daemon. >> > You should use some benchmark which knows how to deal with OS cache, for > example bonnie++ from the ports. > > Use a benchmark with random IO to see just how horrible your RAID-6 > performance will be for random writes. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C348F80.7040706>