From owner-freebsd-questions@FreeBSD.ORG Fri Apr 8 20:41:37 2005 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A177716A4CE; Fri, 8 Apr 2005 20:41:37 +0000 (GMT) Received: from shiva.nextrials.com (shiva.nextrials.com [64.81.74.145]) by mx1.FreeBSD.org (Postfix) with ESMTP id 20B8143D1D; Fri, 8 Apr 2005 20:41:37 +0000 (GMT) (envelope-from dannyman@toldme.com) Received: from [192.168.1.102] (mito.sr.nextrials.com [192.168.1.102]) by shiva.nextrials.com (Postfix) with ESMTP id 906703C2831; Fri, 8 Apr 2005 13:41:36 -0700 (PDT) Message-ID: <4256EC7A.5060107@toldme.com> Date: Fri, 08 Apr 2005 13:41:30 -0700 From: Danny Howard User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050328) X-Accept-Language: en-us, en MIME-Version: 1.0 To: FreeBSD questions , freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: (LONG) ATA Benchmark: 5.x Reads Slower than Writes X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Apr 2005 20:41:37 -0000 Hello, BACKGROUND I need to purchase a new system for our developers, for use as a Postgres database test server. Having a RAID, probably RAID1, is desirable for performance and reliability. I have recently set up a system with a gmirror-based software RAID1 on a pair of 250GB ATA drives. I would like to stick with gmirror, because: - We save money on extra hardware. - Since gmirror is part of FreeBSD, maintenance is a lot easier than with a hardware solution. SUMMARY But before I go balls out, I should see how well it compares to hardware RAID. So, I do some benchmarks with bonnie++. Since this simulates create, write and read on thousands of random files, this sounds like a good approximation of what Postgres does. :) I don't have the time and hardware to do very scientific tests, but I have been able to run a series of benchmarks using bonnie++ on some systems I have available to me. The ATA-based gmirror performs extremely well, compared to a few Adaptec RAIDs that we have, EXCEPT that the sequential and random reads are MUCH SLOWER than the hardware solution, and even *slower than the preceding write operations*. This is counter-intuitive, especially since RAID1 implies slowed writes and faster reads. I tried the benchmark on my workstation (single 2.5" IDE in a laptop) and got comparable write-faster-than-read results. DATA I was able to make use of the following test systems. I ran tests in multi-user, but tried to favor times when there wasn't much background activity: mito: (lone 2.5" ATA) 5.4-PRERELEASE CPU: Intel(R) Pentium(R) M processor 1.50GHz (1495.16-MHz 686-class CPU) atapci0: ata0: channel #0 on atapci0 ata1: channel #1 on atapci0 ad0: 28615MB [58140/16/63] at ata0-master UDMA100 amun: (gmirror RAID1 2 x WD250GB High Intensity) FreeBSD 5.3-RELEASE CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.22-MHz 686-class CPU) atapci0: ata2: channel #0 on atapci0 ata3: channel #1 on atapci0 ad4: 238475MB [484521/16/63] at ata2-master SATA150 ad6: 238475MB [484521/16/63] at ata3-master SATA150 atapci1: acd0: CDROM at ata1-master UDMA33 janus: (Adaptec RAID1 2 x 72G 10,000 RPM SCSI) FreeBSD 4.10-STABLE CPU: Intel(R) Xeon(TM) CPU 2.40GHz (2399.33-MHz 686-class CPU) [DUAL] aac0: mem 0xf8000000-0xfbffffff irq 16 at device 1.0 on pci2 aac0: i960RX 100MHz, 48MB cache memory, optional battery present aac0: Kernel 4.0-0, Build 6011, S/N baec64 aac0: Supported Options=1f7e aacd0: on aac0 aacd0: 69998MB (143357184 sectors) db2: (Adaptec RAID10 4 x 36G 15,000 RPM SCSI) FreeBSD 4.8-STABLE CPU: Intel(R) Xeon(TM) CPU 3.06GHz (3065.81-MHz 686-class CPU) [DUAL] aac0: mem 0xf8000000-0xfbffffff irq 18 at device 2.0 on pci5 aac0: i960RX 100MHz, 48MB cache memory, optional battery present aac0: Kernel 4.0-0, Build 6008, S/N b97ce8 aac0: Supported Options=1f7e aacp0: on aac0 The raw data can be viewed at http://dannyman.toldme.com/scratch/benchmarks/ ANALYSIS Unfortunately, my hardware RAIDs are on FreeBSD 4, and gmirror is on 5. My hardware RAIDs are on dual CPU systems, with 2G RAM, and my gmirror is on a single hyperthreaded CPU with 512M. Yes, sorry, not especially scientific. Maybe the changes in FreeBSD make a big difference? Maybe RAM makes a big difference? The first results show a serious advantage for the gmirror setup: Sequential output (char) gmirror ATA RAID1: avg 320K/s Adaptec SCSI RAID1: avg 222K/s Adaptec SCSI RAID10: avg 202K/s Sequential input (char) gmirror ATA RAID1: avg 617K/s Adaptec SCSI RAID1: avg 345K/s Adaptec SCSI RAID10: avg 336K/s Sequential Output (block) gmirror ATA RAID1: avg 37893K/s Adaptec SCSI RAID1: avg 13829K/s Adaptec SCSI RAID10: avg 40440K/s The gmirror sees slightly poorer performance in random seeks: Rndom Seeks gmirror ATA RAID1: avg 4144/s Adaptec SCSI RAID1: avg 5428/s Adaptec SCSI RAID10: avg 13302/s That all sounds great if I was streaming video, but I want to run a database, opening and closing, reading, writing, and rewriting several small files. This is where things seem to go rotten. We see the ATA performance go to heck on the File Create tests: Sequential Create laptop 2.5" ATA: avg 101/s gmirror ATA RAID1: avg 365/s Adaptec SCSI RAID1: avg 160/s Adaptec SCSI RAID10: avg 412/s Sequential Read laptop 2.5" ATA: avg 76/s # SLOWER than write! gmirror ATA RAID1: avg 251/s # SLOWER than write! Adaptec SCSI RAID1: avg 7862/s Adaptec SCSI RAID10: avg 7618/s Random Create laptop 2.5" ATA: avg 124/s gmirror ATA RAID1: avg 354/s Adaptec SCSI RAID1: avg 155/s Adaptec SCSI RAID10: avg 504/s Random Read laptop 2.5" ATA: avg 57/s # SLOWER than write! gmirror ATA RAID1: avg 144/s # SLOWER than write! Adaptec SCSI RAID1: avg 7655/s Adaptec SCSI RAID10: avg 7413/s CONFUSION Now, I could explain poor read performance by: - Less RAM == Less buffer - Bigger Disks == Slower Seeks - Less CPU == ??? I DO have a 4.8-STABLE with a single IDE disk, no Soft Updates, and faster read than write: Version 1.93c ------Sequential Create------ --------Random Create-------- anubis.xxxxxxxxxxxx -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 10:104884:0/5 183 32 1739 97 502 16 176 32 1624 94 368 13 Latency 707ms 11487us 32824us 488ms 207ms 117ms However, seeing read SLOWER than write ... I have to wonder if something fishy is going on. Suggestions? Ideas? I'm fresh out, at the moment. My suspicion is that something in 5.x is out-of-tune!? Thanks a lot. Sincerely, -danny -- http://dannyman.toldme.com/