From owner-freebsd-performance@FreeBSD.ORG Sun Oct 29 17:20:26 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1077216A412 for ; Sun, 29 Oct 2006 17:20:26 +0000 (UTC) (envelope-from stevep-hv@zpfe.com) Received: from mail.zpfe.com (mail.zpfe.com [208.42.168.33]) by mx1.FreeBSD.org (Postfix) with SMTP id A20B343D46 for ; Sun, 29 Oct 2006 17:20:25 +0000 (GMT) (envelope-from stevep-hv@zpfe.com) Received: (qmail 45767 invoked from network); 29 Oct 2006 17:20:23 -0000 Received: from unknown (HELO revere.zpfe.com) (208.42.168.33) by mail.zpfe.com with SMTP; 29 Oct 2006 17:20:23 -0000 Message-Id: <6.2.3.4.0.20061029095927.05833580@localhost> X-Mailer: QUALCOMM Windows Eudora Version 6.2.3.4 Date: Sun, 29 Oct 2006 11:20:33 -0600 To: Petri Helenius ,performance@freebsd.org From: Steve Peterson In-Reply-To: <4543AD35.30205@he.iki.fi> References: <6.2.3.4.0.20061027180329.020bed68@localhost> <4542D941.2070204@centtech.com> <6.2.3.4.0.20061028124559.02105930@localhost> <4543AD35.30205@he.iki.fi> Mime-Version: 1.0 Content-Type: text/plain; charset="US-ASCII"; format=flowed Cc: Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Oct 2006 17:20:26 -0000 Petri -- thanks for the idea. I ran 2 dds in parallel; they took roughly twice as long in clock time, and had about 1/2 the throughput of the single dd. On my system it doesn't look like how the work is offered to the disk subsystem matters. # time dd if=/dev/zero of=blort1 bs=1m count=1000 & ; time dd if=/dev/zero of=blort2 bs=1m count=1000& [1] 69356 [2] 69357 # 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 382.511089 secs (2741296 bytes/sec) 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 385.677897 secs (2718787 bytes/sec) [2] Done dd if=/dev/zero of=blort2 bs=1m count=1000 0.023u 16.913s 6:25.75 4.3% 20+2813k 3+8772io 2pf+0w [1] + Done dd if=/dev/zero of=blort1 bs=1m count=1000 0.023u 16.880s 6:22.70 4.4% 20+2810k 1+8765io 0pf+0w vmstat output during this run: procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad0 ad4 ad6 ad8 ad10 in sy cs us sy id 1 1 0 101692 13132 0 0 0 0 5872 1857 0 54 62 69 59 2293 321 3914 2 62 35 3 1 0 101692 15576 0 4 0 0 5840 1947 0 52 54 60 57 2023 308 3379 2 64 35 2 1 0 101692 10072 0 0 0 0 5344 0 0 61 54 55 56 2133 368 3496 0 58 42 2 2 0 101692 11496 0 0 0 0 5984 1860 0 61 65 57 61 2377 429 4118 1 69 30 3 0 0 101692 13124 0 0 0 0 5568 1799 0 57 62 62 65 2440 466 4252 1 64 36 1 2 0 101692 12404 0 0 0 0 6176 1868 0 61 70 67 59 1923 482 3337 1 64 36 0 3 0 101692 15080 0 0 0 0 6576 1805 0 67 71 65 68 2423 468 4300 1 68 31 0 2 0 101692 17448 0 0 0 0 5760 1792 0 56 57 57 60 2463 399 4155 4 65 31 That got me thinking about whether there was a bottleneck to the disk somewhere else. I ran a dd reading 1GB simultaneous from each of the 4 disks. Each disk could be read at about 25MB/sec for a combined read transfer rate of 100MB/sec. A snipped of iostat output: tty ad0 ad4 ad6 ad8 ad10 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 334 0.00 0 0.00 127.36 201 24.97 126.73 201 24.85 127.35 198 24.60 126.71 199 24.60 2 0 11 12 75 0 115 0.00 0 0.00 128.00 191 23.86 126.98 188 23.30 128.00 196 24.48 127.36 199 24.73 2 0 15 9 73 0 115 0.00 0 0.00 127.37 203 25.23 127.08 208 25.79 126.75 205 25.35 127.27 175 21.73 2 0 9 9 80 So I'm definitely not getting the full 60MB/sec performance out of the disk with my current controller and mobo. I guess the fundamental question is this -- if I have a 4 disk subsystem that supports an aggregate ~100MB/sec transfer raw to the underlying disks, is it reasonable to expect a ~5MB/sec transfer rate for a RAID5 hosted on that subsystem -- a 95% overhead. Steve At 01:19 PM 10/28/2006, Petri Helenius wrote: >According to my understanding vinum does not overlap requests to >multiple disks when running in raid5 configuration so you're not >going to achieve good numbers with just "single stream" tests. > >Pete > > >Steve Peterson wrote: >>Eric -- thanks for looking at my issue. Here's a dd reading from >>one of the disks underlying the array (the others have basically >>the same performance): >> >>$ time dd if=/dev/ad10 of=/dev/null bs=1m count=1000 >>1000+0 records in >>1000+0 records out >>1048576000 bytes transferred in 15.322421 secs (68434095 bytes/sec) >>0.008u 0.506s 0:15.33 3.2% 20+2715k 0+0io 0pf+0w >> >>and here's a dd reading from the raw gvinum device /dev/gvinum/vol1: >> >>$ time dd if=/dev/gvinum/vol1 of=/dev/null bs=1m count=1000 >>1000+0 records in >>1000+0 records out >>1048576000 bytes transferred in 25.870684 secs (40531437 bytes/sec) >>0.006u 0.552s 0:25.88 2.1% 23+3145k 0+0io 0pf+0w >> >>Is there a way to nondestructively write to the raw disk or gvinum device? >> >>For comparison, here's a read against the raw PATA device on the machine: >> >>$ time dd if=/dev/ad0 of=/dev/null bs=1m count=1000 >>1000+0 records in >>1000+0 records out >>1048576000 bytes transferred in 26.096070 secs (40181376 bytes/sec) >>0.013u 0.538s 0:26.10 2.0% 24+3322k 0+0io 0pf+0w >> >>Steve >> >> >>At 11:14 PM 10/27/2006, Eric Anderson wrote: >>>On 10/27/06 18:03, Steve Peterson wrote: >>>>I recently set up a media server for home use and decided to try >>>>the gvinum raid5 support to learn about it and see how it >>>>performs. It seems slower than I'd expect -- a little under >>>>6MB/second, with about 50 IOs/drive/second -- and I'm trying to >>>>understand why. Any assistance/pointers would be appreciated. >>>>The disk system consists of 4 Seagate NL35 SATA ST3250623NS >>>>drives connected to a Promise TX4300 (PDC40719) controller, >>>>organized as a RAID5 volume via gvinum using this configuration: >>>>drive drive01 device /dev/ad10 >>>>drive drive02 device /dev/ad6 >>>>drive drive03 device /dev/ad4 >>>>drive drive04 device /dev/ad8 >>>>volume vol1 >>>> plex org raid5 256k >>>> sd length 200001m drive drive01 >>>> sd length 200001m drive drive02 >>>> sd length 200001m drive drive03 >>>> sd length 200001m drive drive04 >>>>dd reports the following performance on a 1G file write to the >>>>RAID5 hosted volume: >>>>$ time dd if=/dev/zero of=big.file bs=1m count=1000 >>>>1000+0 records in >>>>1000+0 records out >>>>1048576000 bytes transferred in 179.717742 secs (5834571 bytes/sec) >>>> 179.76 real 0.02 user 16.60 sys >>>>By comparison, creating the same file on the system disk (an old >>>>ATA ST380021A connected via a SIS 730 on the motherboard): >>>>time dd if=/dev/zero of=big.file bs=1m count=1000 >>>>1000+0 records in >>>>1000+0 records out >>>>1048576000 bytes transferred in 28.264056 secs (37099275 bytes/sec) >>>> 28.32 real 0.01 user 19.13 sys >>>>and vmstat reports about 280-300 IOs/second to that drive. >>>>The CPU is pretty weak -- an Athlon 750. Is that the source of >>>>my problem? If you look at the vmstat output below the machine >>>>is busy but not pegged. >>> >>> >>>Try the dd to the raw gvinum device instead of through a >>>filesystem, and also to the individual disks. That will at least >>>tell us where to look. >>> >>> >>>Eric >>> >>> >>> >>>-- >>>------------------------------------------------------------------------ >>>Eric Anderson Sr. Systems Administrator Centaur Technology >>>Anything that works is better than anything that doesn't. >>>------------------------------------------------------------------------ >>>_______________________________________________ >>>freebsd-performance@freebsd.org mailing list >>>http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>>To unsubscribe, send any mail to >>>"freebsd-performance-unsubscribe@freebsd.org" >> >> >>_______________________________________________ >>freebsd-performance@freebsd.org mailing list >>http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>To unsubscribe, send any mail to >>"freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Sun Oct 29 17:26:43 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7173A16A416 for ; Sun, 29 Oct 2006 17:26:43 +0000 (UTC) (envelope-from pete@he.iki.fi) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id C3F7843D53 for ; Sun, 29 Oct 2006 17:26:42 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from localhost (localhost [127.0.0.1]) by silver.he.iki.fi (Postfix) with ESMTP id 2CC61BBFF; Sun, 29 Oct 2006 19:26:41 +0200 (EET) Received: from silver.he.iki.fi ([127.0.0.1]) by localhost (silver.he.iki.fi [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P5+nJqdhWp5w; Sun, 29 Oct 2006 19:26:36 +0200 (EET) Received: from [IPv6:2001:670:84:0:dca1:af59:3d40:6dbd] (unknown [IPv6:2001:670:84:0:dca1:af59:3d40:6dbd]) by silver.he.iki.fi (Postfix) with ESMTP; Sun, 29 Oct 2006 19:26:36 +0200 (EET) Message-ID: <4544E44D.7090906@he.iki.fi> Date: Sun, 29 Oct 2006 19:26:37 +0200 From: Petri Helenius User-Agent: Thunderbird 1.5.0.7 (Windows/20060909) MIME-Version: 1.0 To: Steve Peterson References: <6.2.3.4.0.20061027180329.020bed68@localhost> <4542D941.2070204@centtech.com> <6.2.3.4.0.20061028124559.02105930@localhost> <4543AD35.30205@he.iki.fi> <6.2.3.4.0.20061029095927.05833580@localhost> In-Reply-To: <6.2.3.4.0.20061029095927.05833580@localhost> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: performance@freebsd.org Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Oct 2006 17:26:43 -0000 Steve Peterson wrote: > Petri -- thanks for the idea. > > I ran 2 dds in parallel; they took roughly twice as long in clock > time, and had about 1/2 the throughput of the single dd. On my system > it doesn't look like how the work is offered to the disk subsystem > matters. This is the thing I did with similar results before I abandoned vinum ... The performance from the same disks using either graid3 or a real hardware raid controller is significantly greater. I think there is something in vinum blocking out parallelism. > I guess the fundamental question is this -- if I have a 4 disk > subsystem that supports an aggregate ~100MB/sec transfer raw to the > underlying disks, is it reasonable to expect a ~5MB/sec transfer rate > for a RAID5 hosted on that subsystem -- a 95% overhead. > In my opinion, no. Pete > Steve > > > At 01:19 PM 10/28/2006, Petri Helenius wrote: > >> According to my understanding vinum does not overlap requests to >> multiple disks when running in raid5 configuration so you're not >> going to achieve good numbers with just "single stream" tests. >> >> Pete >> >> >> Steve Peterson wrote: >>> Eric -- thanks for looking at my issue. Here's a dd reading from >>> one of the disks underlying the array (the others have basically the >>> same performance): >>> >>> $ time dd if=/dev/ad10 of=/dev/null bs=1m count=1000 >>> 1000+0 records in >>> 1000+0 records out >>> 1048576000 bytes transferred in 15.322421 secs (68434095 bytes/sec) >>> 0.008u 0.506s 0:15.33 3.2% 20+2715k 0+0io 0pf+0w >>> >>> and here's a dd reading from the raw gvinum device /dev/gvinum/vol1: >>> >>> $ time dd if=/dev/gvinum/vol1 of=/dev/null bs=1m count=1000 >>> 1000+0 records in >>> 1000+0 records out >>> 1048576000 bytes transferred in 25.870684 secs (40531437 bytes/sec) >>> 0.006u 0.552s 0:25.88 2.1% 23+3145k 0+0io 0pf+0w >>> >>> Is there a way to nondestructively write to the raw disk or gvinum >>> device? >>> >>> For comparison, here's a read against the raw PATA device on the >>> machine: >>> >>> $ time dd if=/dev/ad0 of=/dev/null bs=1m count=1000 >>> 1000+0 records in >>> 1000+0 records out >>> 1048576000 bytes transferred in 26.096070 secs (40181376 bytes/sec) >>> 0.013u 0.538s 0:26.10 2.0% 24+3322k 0+0io 0pf+0w >>> >>> Steve >>> >>> >>> At 11:14 PM 10/27/2006, Eric Anderson wrote: >>>> On 10/27/06 18:03, Steve Peterson wrote: >>>>> I recently set up a media server for home use and decided to try >>>>> the gvinum raid5 support to learn about it and see how it >>>>> performs. It seems slower than I'd expect -- a little under >>>>> 6MB/second, with about 50 IOs/drive/second -- and I'm trying to >>>>> understand why. Any assistance/pointers would be appreciated. >>>>> The disk system consists of 4 Seagate NL35 SATA ST3250623NS drives >>>>> connected to a Promise TX4300 (PDC40719) controller, organized as >>>>> a RAID5 volume via gvinum using this configuration: >>>>> drive drive01 device /dev/ad10 >>>>> drive drive02 device /dev/ad6 >>>>> drive drive03 device /dev/ad4 >>>>> drive drive04 device /dev/ad8 >>>>> volume vol1 >>>>> plex org raid5 256k >>>>> sd length 200001m drive drive01 >>>>> sd length 200001m drive drive02 >>>>> sd length 200001m drive drive03 >>>>> sd length 200001m drive drive04 >>>>> dd reports the following performance on a 1G file write to the >>>>> RAID5 hosted volume: >>>>> $ time dd if=/dev/zero of=big.file bs=1m count=1000 >>>>> 1000+0 records in >>>>> 1000+0 records out >>>>> 1048576000 bytes transferred in 179.717742 secs (5834571 bytes/sec) >>>>> 179.76 real 0.02 user 16.60 sys >>>>> By comparison, creating the same file on the system disk (an old >>>>> ATA ST380021A connected via a SIS 730 on the motherboard): >>>>> time dd if=/dev/zero of=big.file bs=1m count=1000 >>>>> 1000+0 records in >>>>> 1000+0 records out >>>>> 1048576000 bytes transferred in 28.264056 secs (37099275 bytes/sec) >>>>> 28.32 real 0.01 user 19.13 sys >>>>> and vmstat reports about 280-300 IOs/second to that drive. >>>>> The CPU is pretty weak -- an Athlon 750. Is that the source of my >>>>> problem? If you look at the vmstat output below the machine is >>>>> busy but not pegged. >>>> >>>> >>>> Try the dd to the raw gvinum device instead of through a >>>> filesystem, and also to the individual disks. That will at least >>>> tell us where to look. >>>> >>>> >>>> Eric >>>> >>>> >>>> >>>> -- >>>> ------------------------------------------------------------------------ >>>> >>>> Eric Anderson Sr. Systems Administrator Centaur >>>> Technology >>>> Anything that works is better than anything that doesn't. >>>> ------------------------------------------------------------------------ >>>> >>>> _______________________________________________ >>>> freebsd-performance@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>>> To unsubscribe, send any mail to >>>> "freebsd-performance-unsubscribe@freebsd.org" >>> >>> >>> _______________________________________________ >>> freebsd-performance@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>> To unsubscribe, send any mail to >>> "freebsd-performance-unsubscribe@freebsd.org" >> > > > From owner-freebsd-performance@FreeBSD.ORG Sun Oct 29 17:59:36 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6AA9816A415 for ; Sun, 29 Oct 2006 17:59:36 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from multiplay.co.uk (core6.multiplay.co.uk [85.236.96.23]) by mx1.FreeBSD.org (Postfix) with ESMTP id A7A2743D68 for ; Sun, 29 Oct 2006 17:59:32 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([85.236.96.60]) by multiplay.co.uk (multiplay.co.uk [85.236.96.23]) (MDaemon PRO v9.0.1) with ESMTP id md50003148384.msg for ; Sun, 29 Oct 2006 17:58:55 +0000 Message-ID: <003f01c6fb83$e4b96280$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Petri Helenius" , , "Steve Peterson" References: <6.2.3.4.0.20061027180329.020bed68@localhost><4542D941.2070204@centtech.com><6.2.3.4.0.20061028124559.02105930@localhost><4543AD35.30205@he.iki.fi> <6.2.3.4.0.20061029095927.05833580@localhost> Date: Sun, 29 Oct 2006 17:58:44 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2869 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2962 X-Spam-Processed: multiplay.co.uk, Sun, 29 Oct 2006 17:58:55 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 85.236.96.60 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Sun, 29 Oct 2006 17:58:55 +0000 Cc: Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Oct 2006 17:59:36 -0000 Steve Peterson wrote: > I guess the fundamental question is this -- if I have a 4 disk > subsystem that supports an aggregate ~100MB/sec transfer raw to the > underlying disks, is it reasonable to expect a ~5MB/sec transfer rate > for a RAID5 hosted on that subsystem -- a 95% overhead. Absolutely not, when I was investigating RAID5 performance the quickest option I found was linux's software RAID which gave performance of 270MB/s iirc across 5 * 400Gb Disks on a Highpoint 1820a. While doing this I gave givnum a quick test and saw similar results as you so gave up with it as I didnt have time to investigate further. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon Oct 30 01:32:14 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5D8AD16A40F for ; Mon, 30 Oct 2006 01:32:14 +0000 (UTC) (envelope-from grog@lemis.com) Received: from ext-gw.lemis.com (ext-gw.lemis.com [150.101.14.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8BBF943D55 for ; Mon, 30 Oct 2006 01:32:13 +0000 (GMT) (envelope-from grog@lemis.com) Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.135]) by ext-gw.lemis.com (Postfix) with ESMTP id 56D851330F3; Mon, 30 Oct 2006 12:02:11 +1030 (CST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id 38B8F9C277; Mon, 30 Oct 2006 12:02:11 +1030 (CST) Date: Mon, 30 Oct 2006 12:02:11 +1030 From: Greg 'groggy' Lehey To: Petri Helenius Message-ID: <20061030013211.GC1052@wantadilla.lemis.com> References: <6.2.3.4.0.20061027180329.020bed68@localhost> <4542D941.2070204@centtech.com> <6.2.3.4.0.20061028124559.02105930@localhost> <4543AD35.30205@he.iki.fi> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="f0KYrhQ4vYSV2aJu" Content-Disposition: inline In-Reply-To: <4543AD35.30205@he.iki.fi> User-Agent: Mutt/1.4.2.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 VoIP: sip:0871270137@sip.internode.on.net WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 Cc: Steve Peterson , performance@freebsd.org Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2006 01:32:14 -0000 --f0KYrhQ4vYSV2aJu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Saturday, 28 October 2006 at 22:19:17 +0300, Petri Helenius wrote: > > According to my understanding vinum does not overlap requests to > multiple disks when running in raid5 configuration Yes, it does. I suspect that gvinum does too. > so you're not going to achieve good numbers with just "single > stream" tests. Single stream tests aren't very good examples for RAID-5, because it performs writes in two steps: first it reads the old data, then it writes the new data. Greg [trailing junk omitted] -- See complete headers for address and phone numbers. --f0KYrhQ4vYSV2aJu Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFFRVYbIubykFB6QiMRAgqyAJ9vWtgQkDf2hncrSNu/bQLtBvsbsQCfaYki /5vq36mnqiI8TMseXtc2qfk= =hJ3d -----END PGP SIGNATURE----- --f0KYrhQ4vYSV2aJu-- From owner-freebsd-performance@FreeBSD.ORG Mon Oct 30 01:33:17 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B8E5016A407 for ; Mon, 30 Oct 2006 01:33:17 +0000 (UTC) (envelope-from grog@lemis.com) Received: from ext-gw.lemis.com (ext-gw.lemis.com [150.101.14.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7BDC343D46 for ; Mon, 30 Oct 2006 01:33:16 +0000 (GMT) (envelope-from grog@lemis.com) Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.135]) by ext-gw.lemis.com (Postfix) with ESMTP id EBF161330F3; Mon, 30 Oct 2006 12:03:14 +1030 (CST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id D4DA19C277; Mon, 30 Oct 2006 12:03:14 +1030 (CST) Date: Mon, 30 Oct 2006 12:03:14 +1030 From: Greg 'groggy' Lehey To: Steve Peterson Message-ID: <20061030013314.GD1052@wantadilla.lemis.com> References: <6.2.3.4.0.20061027180329.020bed68@localhost> <4542D941.2070204@centtech.com> <6.2.3.4.0.20061028124559.02105930@localhost> <4543AD35.30205@he.iki.fi> <6.2.3.4.0.20061029095927.05833580@localhost> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/3yNEOqWowh/8j+e" Content-Disposition: inline In-Reply-To: <6.2.3.4.0.20061029095927.05833580@localhost> User-Agent: Mutt/1.4.2.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 VoIP: sip:0871270137@sip.internode.on.net WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 Cc: performance@freebsd.org, Petri Helenius Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2006 01:33:17 -0000 --/3yNEOqWowh/8j+e Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Sunday, 29 October 2006 at 11:20:33 -0600, Steve Peterson wrote: > Petri -- thanks for the idea. It would be a good idea to quote it. Following this thread is almost impossible. > I ran 2 dds in parallel; they took roughly twice as long in clock > time, and had about 1/2 the throughput of the single dd. On my > system it doesn't look like how the work is offered to the disk > subsystem matters. Try running rawio. That's what it's for. Greg [more trailing junk omitted. Why do you attach it?] -- See complete headers for address and phone numbers. --/3yNEOqWowh/8j+e Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFFRVZaIubykFB6QiMRAsYsAJ0Y0xdUK2eQNw/VqRdtUqwcRBm8ngCeISaB 8IibjbOZY9oaqQV3nGXI1kE= =7D+4 -----END PGP SIGNATURE----- --/3yNEOqWowh/8j+e-- From owner-freebsd-performance@FreeBSD.ORG Mon Oct 30 05:11:33 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9FCF016A416; Mon, 30 Oct 2006 05:11:33 +0000 (UTC) (envelope-from pete@he.iki.fi) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2744143D58; Mon, 30 Oct 2006 05:11:32 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from localhost (localhost [127.0.0.1]) by silver.he.iki.fi (Postfix) with ESMTP id 88062BBFF; Mon, 30 Oct 2006 07:11:30 +0200 (EET) Received: from silver.he.iki.fi ([127.0.0.1]) by localhost (silver.he.iki.fi [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Flw+ekcJxhyE; Mon, 30 Oct 2006 07:11:28 +0200 (EET) Received: from [IPv6:2001:670:84:0:dca1:af59:3d40:6dbd] (unknown [IPv6:2001:670:84:0:dca1:af59:3d40:6dbd]) by silver.he.iki.fi (Postfix) with ESMTP; Mon, 30 Oct 2006 07:11:27 +0200 (EET) Message-ID: <45458981.8010302@he.iki.fi> Date: Mon, 30 Oct 2006 07:11:29 +0200 From: Petri Helenius User-Agent: Thunderbird 1.5.0.7 (Windows/20060909) MIME-Version: 1.0 To: Greg 'groggy' Lehey References: <6.2.3.4.0.20061027180329.020bed68@localhost> <4542D941.2070204@centtech.com> <6.2.3.4.0.20061028124559.02105930@localhost> <4543AD35.30205@he.iki.fi> <20061030013211.GC1052@wantadilla.lemis.com> In-Reply-To: <20061030013211.GC1052@wantadilla.lemis.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Steve Peterson , performance@freebsd.org Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2006 05:11:33 -0000 Greg 'groggy' Lehey wrote: > > Single stream tests aren't very good examples for RAID-5, because it > performs writes in two steps: first it reads the old data, then it > writes the new data. > If it really does it this way, instead doing write-only when writing sufficiently large blocks, that would explain the performance due to double rotational latency wait for each stripe size. Most implementations also use read/write caches to optimize this even further. Pete From owner-freebsd-performance@FreeBSD.ORG Mon Oct 30 05:19:24 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5128616A412 for ; Mon, 30 Oct 2006 05:19:24 +0000 (UTC) (envelope-from grog@lemis.com) Received: from ext-gw.lemis.com (ext-gw.lemis.com [150.101.14.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id F0DB243D49 for ; Mon, 30 Oct 2006 05:19:22 +0000 (GMT) (envelope-from grog@lemis.com) Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.135]) by ext-gw.lemis.com (Postfix) with ESMTP id C82B91323F1; Mon, 30 Oct 2006 15:49:13 +1030 (CST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id B48CA9C277; Mon, 30 Oct 2006 15:49:13 +1030 (CST) Date: Mon, 30 Oct 2006 15:49:13 +1030 From: Greg 'groggy' Lehey To: Petri Helenius Message-ID: <20061030051913.GG1052@wantadilla.lemis.com> References: <6.2.3.4.0.20061027180329.020bed68@localhost> <4542D941.2070204@centtech.com> <6.2.3.4.0.20061028124559.02105930@localhost> <4543AD35.30205@he.iki.fi> <20061030013211.GC1052@wantadilla.lemis.com> <45458981.8010302@he.iki.fi> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="xjyYRNSh/RebjC6o" Content-Disposition: inline In-Reply-To: <45458981.8010302@he.iki.fi> User-Agent: Mutt/1.4.2.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 VoIP: sip:0871270137@sip.internode.on.net WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 Cc: Steve Peterson , performance@freebsd.org Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2006 05:19:24 -0000 --xjyYRNSh/RebjC6o Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Monday, 30 October 2006 at 7:11:29 +0200, Petri Helenius wrote: > Greg 'groggy' Lehey wrote: >> >> Single stream tests aren't very good examples for RAID-5, because it >> performs writes in two steps: first it reads the old data, then it >> writes the new data. > > If it really does it this way, instead doing write-only when writing > sufficiently large blocks, that would explain the performance due to > double rotational latency wait for each stripe size. No, this works at the bio layer, where everything is cushioned on buffer cache. > Most implementations also use read/write caches to optimize this > even further. "Sufficiently large data blocks" equates to several megabytes. Currently MAXPHYS, the largest transfer request that would get to the bio layer, is 131072 bytes. This would imply a stripe size of not more than 32 kB for a five disk array, which is unrealistically small. I did consider this optimization, but it would only work if Vinum were first to buffer multiple requests, and there are all sorts of reliability issues there. For example, you'd have to lie about the first few requests that were only buffered and not actually sent out to disk. Possibly it should be done anyway. Greg -- See complete headers for address and phone numbers. --xjyYRNSh/RebjC6o Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFFRYtRIubykFB6QiMRAsL3AJ93edVFQOlzRExsHe3X7nsqLn3qqQCfanSO Ri9z3PrrDgWp2j/O8+yMhCM= =opGf -----END PGP SIGNATURE----- --xjyYRNSh/RebjC6o-- From owner-freebsd-performance@FreeBSD.ORG Mon Oct 30 07:05:33 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F1AB416A40F for ; Mon, 30 Oct 2006 07:05:33 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30309.mail.mud.yahoo.com (web30309.mail.mud.yahoo.com [209.191.69.71]) by mx1.FreeBSD.org (Postfix) with SMTP id 76EB443D55 for ; Mon, 30 Oct 2006 07:05:33 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 5497 invoked by uid 60001); 30 Oct 2006 07:05:32 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=AMTuyXRl7hnVMy2acVnByDac81FVHd/jd8fEeNYI4Gi1/OelGadM5MK+0S/sS+BMYp1Zie9HKk0IcOUKVFpzrXJRA4adtbaix9I66Q2qsHWuznhn01hpfb44rvDS8Ch3JPHj/ocO863zOIJ7w5yH1eIJBVpWi4VEb0rIpDsz3i8= ; Message-ID: <20061030070532.5495.qmail@web30309.mail.mud.yahoo.com> Received: from [213.54.132.43] by web30309.mail.mud.yahoo.com via HTTP; Sun, 29 Oct 2006 23:05:32 PST Date: Sun, 29 Oct 2006 23:05:32 -0800 (PST) From: "R. B. Riddick" To: performance@freebsd.org In-Reply-To: <20061030051913.GG1052@wantadilla.lemis.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2006 07:05:34 -0000 --- Greg 'groggy' Lehey wrote: > "Sufficiently large data blocks" equates to several megabytes. > Currently MAXPHYS, the largest transfer request that would get to the > bio layer, is 131072 bytes. This would imply a stripe size of not > more than 32 kB for a five disk array, which is unrealistically small. > > I did consider this optimization, but it would only work if Vinum were > first to buffer multiple requests, and there are all sorts of > reliability issues there. For example, you'd have to lie about the > first few requests that were only buffered and not actually sent out > to disk. Possibly it should be done anyway. > I did it that way in my graid5 class: http://home.tiscali.de/cmdr_faako/geom_raid5.tbz The source might look somehow incomprehensible, but it seems to be quite stable (on amd64 and i386)... I tried to benchmark with a 3 disk array: Single disk: 30MB/sec write speed graid5 device: 20MB/sec write speed I would be interested in other benchmarks about it. -Arne ____________________________________________________________________________________ Get your email and see which of your friends are online - Right on the New Yahoo.com (http://www.yahoo.com/preview) From owner-freebsd-performance@FreeBSD.ORG Mon Oct 30 07:08:12 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8D90E16A40F for ; Mon, 30 Oct 2006 07:08:12 +0000 (UTC) (envelope-from grog@lemis.com) Received: from ext-gw.lemis.com (ext-gw.lemis.com [150.101.14.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id A8F2943D6D for ; Mon, 30 Oct 2006 07:08:11 +0000 (GMT) (envelope-from grog@lemis.com) Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.135]) by ext-gw.lemis.com (Postfix) with ESMTP id 72F94133129; Mon, 30 Oct 2006 17:38:09 +1030 (CST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id 5ECD49BF13; Mon, 30 Oct 2006 17:38:09 +1030 (CST) Date: Mon, 30 Oct 2006 17:38:09 +1030 From: Greg 'groggy' Lehey To: "R. B. Riddick" Message-ID: <20061030070809.GJ1052@wantadilla.lemis.com> References: <20061030051913.GG1052@wantadilla.lemis.com> <20061030070532.5495.qmail@web30309.mail.mud.yahoo.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4vpci17Ql0Nrbul2" Content-Disposition: inline In-Reply-To: <20061030070532.5495.qmail@web30309.mail.mud.yahoo.com> User-Agent: Mutt/1.4.2.1i Organization: The FreeBSD Project Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 VoIP: sip:0871270137@sip.internode.on.net WWW-Home-Page: http://www.FreeBSD.org/ X-PGP-Fingerprint: 9A1B 8202 BCCE B846 F92F 09AC 22E6 F290 507A 4223 Cc: performance@freebsd.org Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2006 07:08:12 -0000 --4vpci17Ql0Nrbul2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Sunday, 29 October 2006 at 23:05:32 -0800, R. B. Riddick wrote: > --- Greg 'groggy' Lehey wrote: >> "Sufficiently large data blocks" equates to several megabytes. >> Currently MAXPHYS, the largest transfer request that would get to the >> bio layer, is 131072 bytes. This would imply a stripe size of not >> more than 32 kB for a five disk array, which is unrealistically small. >> >> I did consider this optimization, but it would only work if Vinum were >> first to buffer multiple requests, and there are all sorts of >> reliability issues there. For example, you'd have to lie about the >> first few requests that were only buffered and not actually sent out >> to disk. Possibly it should be done anyway. >> > I did it that way in my graid5 class: > http://home.tiscali.de/cmdr_faako/geom_raid5.tbz I would have taken a look at it if the sources had been directly web viewable. > The source might look somehow incomprehensible, Heh. I know the feeling :-) > but it seems to be quite stable (on amd64 and i386)... > > > I tried to benchmark with a 3 disk array: > Single disk: 30MB/sec write speed > graid5 device: 20MB/sec write speed Can you give more details? Stripe size, request size, number of concurrent accessors, etc.? > I would be interested in other benchmarks about it. Try rawio. Greg -- See complete headers for address and phone numbers. --4vpci17Ql0Nrbul2 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFFRaTZIubykFB6QiMRArD6AJ4ygV+/v2wMI9+pOnBAWDYE9yecPACfTA18 0K8V4QRRIS1wG+qq14zlb4k= =R5eA -----END PGP SIGNATURE----- --4vpci17Ql0Nrbul2-- From owner-freebsd-performance@FreeBSD.ORG Mon Oct 30 08:01:36 2006 Return-Path: X-Original-To: performance@freebsd.org Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 866CE16A47B for ; Mon, 30 Oct 2006 08:01:36 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30314.mail.mud.yahoo.com (web30314.mail.mud.yahoo.com [209.191.69.76]) by mx1.FreeBSD.org (Postfix) with SMTP id 86F7943D5A for ; Mon, 30 Oct 2006 08:01:35 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 79747 invoked by uid 60001); 30 Oct 2006 08:01:35 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=DjMQO3Y8MjmSjdtZ0aRwPU8X7dhOi9EizCcGr2fS9REEHGxKBQgKaS6YdXl2vJA29kbqk9BPHdeyC9tUtDk5L/JdUUWavPLO8k8z8K76Ao3hWHxmbmfEj2XfOs69qo3OHLGYGLHxSVMdSAlM1vAUEmU2YIEn3fWL3qMRo21ONek= ; Message-ID: <20061030080135.79745.qmail@web30314.mail.mud.yahoo.com> Received: from [213.54.132.43] by web30314.mail.mud.yahoo.com via HTTP; Mon, 30 Oct 2006 00:01:34 PST Date: Mon, 30 Oct 2006 00:01:34 -0800 (PST) From: "R. B. Riddick" To: Greg 'groggy' Lehey In-Reply-To: <20061030070809.GJ1052@wantadilla.lemis.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: performance@freebsd.org Subject: Re: gvinum raid5 performance seems slow X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2006 08:01:36 -0000 --- Greg 'groggy' Lehey wrote: > On Sunday, 29 October 2006 at 23:05:32 -0800, R. B. Riddick wrote: > > I did it that way in my graid5 class: > > http://home.tiscali.de/cmdr_faako/geom_raid5.tbz > > I would have taken a look at it if the sources had been directly web > viewable. > Hmm... Too lazy to do that... > > I tried to benchmark with a 3 disk array: > > Single disk: 30MB/sec write speed > > graid5 device: 20MB/sec write speed > > Can you give more details? Stripe size, request size, number of > concurrent accessors, etc.? > On a UFS. 128KB stripe size, a single dd with bs=1m. UDMA100 (one disk of the graid5 does UDMA133). > > I would be interested in other benchmarks about it. > > Try rawio. > Cant use rawio, because I already have a productive UFS on my only graid5 (no more space left)... Bonnie says (on UFS on single disk): -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 2000 37628 46.1 34951 19.1 13618 7.8 35444 45.2 38598 14.5 102.4 0.9 With 2 concurrent bonnies: -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 200 13143 15.7 32817 18.2 3037 1.9 27621 35.0 299571 93.3 18556.1 90.8 200 13112 16.0 32524 17.7 3396 2.1 53843 67.0 196843 66.1 205.1 1.3 And on UFS on graid5 device (128KB stripe size, 3 disks): -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 2000 24671 32.2 26300 16.9 11877 6.7 17502 22.7 39775 15.0 115.2 0.8 With 2 concurrent bonnies: -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 200 4568 6.1 19114 11.1 1906 1.1 62400 76.8 191310 62.5 3802.2 23.3 200 4569 6.1 16044 10.4 1707 1.0 48317 57.9 275623 86.4 17576.5 85.7 -Arne ____________________________________________________________________________________ Access over 1 million songs - Yahoo! Music Unlimited (http://music.yahoo.com/unlimited) From owner-freebsd-performance@FreeBSD.ORG Mon Oct 30 19:34:08 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 46BD716A4C2 for ; Mon, 30 Oct 2006 19:34:08 +0000 (UTC) (envelope-from marcelo@registro.br) Received: from clone.registro.br (clone.registro.br [200.160.2.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 68B2244000 for ; Mon, 30 Oct 2006 19:27:05 +0000 (GMT) (envelope-from marcelo@registro.br) Received: by clone.registro.br (Postfix, from userid 1014) id 250742A4F5; Mon, 30 Oct 2006 16:27:02 -0300 (BRT) Date: Mon, 30 Oct 2006 16:27:02 -0300 From: Marcelo Gardini do Amaral To: freebsd-performance@freebsd.org Message-ID: <20061030192702.GG76994@registro.br> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-Mailman-Approved-At: Tue, 31 Oct 2006 00:45:48 +0000 Cc: kreios@gmail.com Subject: DNS Performance Numbers X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Oct 2006 19:34:08 -0000 Dave, could you please describe you test set? I've posted some results months ago and they were kind different. I have done some tests [1] [2] with bind and queryperf and my result on FreeBSD 6.1 was very poor if compared with 4.11. Besides this, I have had bad results with NSD + 6.1 too and it doesn't use threads. Because of that I realized that there is something strange with UDP traffic on 6.1, as it was discussed in the threads below. [1] http://lists.freebsd.org/pipermail/freebsd-stable/2006-September/028247.html [2] http://lists.freebsd.org/pipermail/freebsd-net/2006-September/011767.html -- Att., Marcelo Gardini From owner-freebsd-performance@FreeBSD.ORG Tue Oct 31 04:35:31 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9898B16A407 for ; Tue, 31 Oct 2006 04:35:31 +0000 (UTC) (envelope-from kreios@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 27C6D43D4C for ; Tue, 31 Oct 2006 04:35:31 +0000 (GMT) (envelope-from kreios@gmail.com) Received: by wx-out-0506.google.com with SMTP id i27so1476080wxd for ; Mon, 30 Oct 2006 20:35:30 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:in-reply-to:references:mime-version:content-type:message-id:cc:content-transfer-encoding:from:subject:date:to:x-mailer; b=mBe5lTSSAbMXdpXC6Lz7SoD+HImQnhC2IDmpXTGCMjCh76XujCutfqP2QE1Yr9s1JlyskxMsOn5LthpkTnaPMBmRKTUv09tev2kHuzVmjLw8qp81djwJk4u0vp08mVOUPEpOv1caRsElXPZDyv1StrBctkgGYp963LctXzlfr7E= Received: by 10.70.65.8 with SMTP id n8mr6402987wxa; Mon, 30 Oct 2006 20:35:30 -0800 (PST) Received: from ?192.168.1.198? ( [165.91.48.122]) by mx.google.com with ESMTP id 43sm6040179wri.2006.10.30.20.35.29; Mon, 30 Oct 2006 20:35:30 -0800 (PST) In-Reply-To: <20061030192702.GG76994@registro.br> References: <20061030192702.GG76994@registro.br> Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <9D15CAB5-0737-4773-9357-335E16A7A4B8@gmail.com> Content-Transfer-Encoding: 7bit From: Dave Date: Mon, 30 Oct 2006 22:35:26 -0600 To: Marcelo Gardini do Amaral X-Mailer: Apple Mail (2.752.2) Cc: freebsd-performance@freebsd.org Subject: Re: DNS Performance Numbers X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 Oct 2006 04:35:31 -0000 On Oct 30, 2006, at 1:27 PM, Marcelo Gardini do Amaral wrote: > Dave, could you please describe you test set? All the hardware is the same. Queryperf was run on a FreeBSD 6.1 box. Both boxes are connected through a Cisco switch. All connections are gig, full duplex. Test was done with changing the OS on the DNS server box. Zone data is a small zone of 256 hosts. queryperf command is: queryperf -s dnstest -d small.dat -q 40 -l 60 > I've posted some results months ago and they were kind different. > > I have done some tests [1] [2] with bind and queryperf and my result > on FreeBSD 6.1 was very poor if compared with 4.11. > > Besides this, I have had bad results with NSD + 6.1 too and it doesn't > use threads. Because of that I realized that there is something > strange with UDP traffic on 6.1, as it was discussed in the threads > below. > > [1] > http://lists.freebsd.org/pipermail/freebsd-stable/2006-September/ > 028247.html > > [2] > http://lists.freebsd.org/pipermail/freebsd-net/2006-September/ > 011767.html I am still in the mist of testing but re-reading your posts, I figured I would try a few tests with linux as the queryperf host. A task on the list but which I haven't gotten to yet. The results are interesting. I would also take them with a grain of salt. This was done very quickly to see what initial numbers would pop out. Version of FreeBSD is 6.1. named was compiled with threads and using libthr. Polling was not enabled. query OS named OS qps ------------------------------ FreeBSD FreeBSD 38442 Linux FreeBSD 22211 Linux Linux 53225 FreeBSD 4.11 is not easily installed on these boxes and currently means that I have to physically touch them so I do not know when I will have time to try it out as a query host. -- DaveD From owner-freebsd-performance@FreeBSD.ORG Sat Nov 4 20:56:06 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 359F016A403 for ; Sat, 4 Nov 2006 20:56:06 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6245843D69 for ; Sat, 4 Nov 2006 20:55:57 +0000 (GMT) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id E091148803; Sat, 4 Nov 2006 21:55:54 +0100 (CET) Received: from localhost (dll2.neoplus.adsl.tpnet.pl [83.24.41.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 2AA5C48800; Sat, 4 Nov 2006 21:55:48 +0100 (CET) Date: Sat, 4 Nov 2006 21:55:42 +0100 From: Pawel Jakub Dawidek To: Oles Hnatkevych Message-ID: <20061104205542.GB83118@garage.freebsd.pl> References: <184686318.20061104222857@able.com.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ZoaI/ZTpAVc4A5k6" Content-Disposition: inline In-Reply-To: <184686318.20061104222857@able.com.ua> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: ** X-Spam-Status: No, score=2.6 required=3.0 tests=BAYES_00,RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL,RCVD_IN_XBL autolearn=no version=3.0.4 Cc: freebsd-performance@freebsd.org Subject: Re: geom stripe perfomance question X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Nov 2006 20:56:06 -0000 --ZoaI/ZTpAVc4A5k6 Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Nov 04, 2006 at 10:28:57PM +0200, Oles Hnatkevych wrote: > Hello all! >=20 > I wonder why geom stripe works much worse than the separate disks that > constitute stripe. >=20 > I have a stripe from two disks. Disks are on separate ATA channels. >=20 > root# gstripe list > Geom name: bigdata > State: UP > Status: Total=3D2, Online=3D2 > Type: AUTOMATIC > Stripesize: 262144 > ID: 4233231544 > Providers: > 1. Name: stripe/bigdata > Mediasize: 500118323200 (466G) > Sectorsize: 512 > Mode: r1w1e1 > Consumers: > 1. Name: ad1 > Mediasize: 250059350016 (233G) > Sectorsize: 512 > Mode: r1w1e2 > Number: 0 > 2. Name: ad2 > Mediasize: 250059350016 (233G) > Sectorsize: 512 > Mode: r1w1e2 > Number: 1 >=20 > Now let's read one of them and stripe. >=20 > root# dd if=3D/dev/ad1 of=3D/dev/null bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 14.579483 secs (71921343 bytes/sec) >=20 > root# dd if=3D/dev/stripe/bigdata of=3D/dev/null bs=3D1m count=3D1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 15.882796 secs (66019610 bytes/sec) >=20 > What I would expect is doubling the speed of transfer, not > slowing down. Am I wrong? Or is geom_stripe inefficient? >=20 > I am running 6.2-PRERELEASE on Celeron 800. It should be, but first try something like: dd if=3D/dev/ad1 of=3D/dev/null bs=3D1m count=3D1000 &; dd if=3D/dev/ad2 of= =3D/dev/null bs=3D1m count=3D1000 & --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --ZoaI/ZTpAVc4A5k6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (FreeBSD) iD8DBQFFTP5OForvXbEpPzQRAhQXAKCsV/R15GqWtK57wItzNVsgK2MGLACgw57o oG5a8XnIkwlOaB1ftEBgdA4= =yDHV -----END PGP SIGNATURE----- --ZoaI/ZTpAVc4A5k6-- From owner-freebsd-performance@FreeBSD.ORG Sat Nov 4 21:00:42 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A27AD16A589 for ; Sat, 4 Nov 2006 21:00:42 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.FreeBSD.org (Postfix) with ESMTP id 571D743D5A for ; Sat, 4 Nov 2006 21:00:10 +0000 (GMT) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 55A5D487F3; Sat, 4 Nov 2006 22:00:09 +0100 (CET) Received: from localhost (dll2.neoplus.adsl.tpnet.pl [83.24.41.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id C363345681 for ; Sat, 4 Nov 2006 22:00:03 +0100 (CET) Date: Sat, 4 Nov 2006 21:59:58 +0100 From: Pawel Jakub Dawidek To: freebsd-performance@freebsd.org Message-ID: <20061104205957.GC83118@garage.freebsd.pl> References: <184686318.20061104222857@able.com.ua> <20061104205542.GB83118@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="xo44VMWPx7vlQ2+2" Content-Disposition: inline In-Reply-To: <20061104205542.GB83118@garage.freebsd.pl> X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 User-Agent: mutt-ng/devel-r804 (FreeBSD) X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: ** X-Spam-Status: No, score=2.6 required=3.0 tests=BAYES_00,RCVD_IN_NJABL_DUL, RCVD_IN_SORBS_DUL,RCVD_IN_XBL autolearn=no version=3.0.4 Subject: Re: geom stripe perfomance question X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Nov 2006 21:00:42 -0000 --xo44VMWPx7vlQ2+2 Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Nov 04, 2006 at 09:55:42PM +0100, Pawel Jakub Dawidek wrote: > On Sat, Nov 04, 2006 at 10:28:57PM +0200, Oles Hnatkevych wrote: [...] Fix your mail server. : host mail.able.com.ua[80.91.162.66] said: 554 5.7.1 : Client host rejected: Spammers must die (in reply to RCPT TO command) My mail server is not sending spam and has static IP address, really. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --xo44VMWPx7vlQ2+2 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.4 (FreeBSD) iD8DBQFFTP9NForvXbEpPzQRArVjAJ4pzJpQGEENvaR2w0epYcD83OvbPACg5Lb/ yaR0oDYivQhfTokiQ2DlAOM= =o9D2 -----END PGP SIGNATURE----- --xo44VMWPx7vlQ2+2--