From owner-freebsd-performance@FreeBSD.ORG Sun May 1 10:34:01 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6DF2516A4CE for ; Sun, 1 May 2005 10:34:01 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 28F7843D45 for ; Sun, 1 May 2005 10:34:01 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id BC0AE46B09; Sun, 1 May 2005 06:34:00 -0400 (EDT) Date: Sun, 1 May 2005 11:36:58 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Arne WXrner In-Reply-To: <20050430211230.20778.qmail@web41203.mail.yahoo.com> Message-ID: <20050501112902.S66519@fledge.watson.org> References: <20050430211230.20778.qmail@web41203.mail.yahoo.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: freebsd-performance@freebsd.org cc: Petri Helenius Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2005 10:34:01 -0000 On Sat, 30 Apr 2005, Arne WXrner wrote: > 3. The man page geom(4) of R5.3 says "The GEOM framework > provides an infrastructure in which "classes" can per- > form transformations on disk I/O requests on their path > from the upper kernel to the device drivers and back. > > Could it be, that geom slows something down (in some boxes the reading > ops are very slow; in my box the writing ops are very slow)? There are three types of overhead associated with GEOM, some of which existed in 4.x also, just not under the name "GEOM". Some can be easily characterized through benchmarking just on 5.x, other bits cannot. Here they are: (1) Fixed overhead per-transaction of entering and leaving the GEOM framework. Because this involves context switches and queueing, this overhead can be amortized under high transaction rates. (2) Cost of entering each "GEOM module" as part of the framework, or costs assocated with any GEOM module you might run, which typically involves allocating a bio, as well as queueing operations. (3) Cost of specific GEOM modules, such as transforms, RAID, etc -- may include computation, scatter/gather of small I/Os into larger ones, etc. However, it's worth noting that GEOM also introdues performance benefits, such as create a clean hand-off separation between the file system code and the device code, so that MPSAFE devices can interact safely with non-MPSAFE file systems, and in 6.x, MPSAFE file systems can interact safely wit non-MPSAFE storage devices. It also permits parallelism -- various bits of storage processing and handling can be running on a separate CPU from a file system generating a set of synchronous I/O's. One interesting set of micro-benchmarks to identify the incremental costs of (2) and (3) is to run identical I/O transations against the same regions of physical disk using different layers in the partition stack. I.e., against a region of ad0s1a, against an offset region on ad0s1, and against a further offset region of ad0. If they're against the same bits of disk, the main difference here will be the additional processing of the layers in the stack. A little bit of math is required to figure out the offset, but dd should be usable to figure out the incremental cost. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Sun May 1 11:23:51 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9D0EE16A4CE for ; Sun, 1 May 2005 11:23:51 +0000 (GMT) Received: from web41205.mail.yahoo.com (web41205.mail.yahoo.com [66.218.93.38]) by mx1.FreeBSD.org (Postfix) with SMTP id 52B4343D45 for ; Sun, 1 May 2005 11:23:51 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 4186 invoked by uid 60001); 1 May 2005 11:23:51 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=nYxPKv9wHaWywpjdlEszERk9jqhrnHt9L7C95xzKnKZq3jPpyTDxh3yyksG1iDD111xEtqQiNe3BYYHYMErbBIrpUC2icKypykN50bx2FMx+bM7kySTDok4pE9Vx24uq9nCcTuE6zcErXqLAfi6e0c9n7q+6kFcFC0TJUBRps3g= ; Message-ID: <20050501112351.4184.qmail@web41205.mail.yahoo.com> Received: from [83.129.191.33] by web41205.mail.yahoo.com via HTTP; Sun, 01 May 2005 04:23:51 PDT Date: Sun, 1 May 2005 04:23:51 -0700 (PDT) From: Arne "Wörner" To: Robert Watson In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2005 11:23:51 -0000 --- Robert Watson wrote: > On Sat, 30 Apr 2005, Arne WXrner wrote: > > 3. The man page geom(4) of R5.3 says "The GEOM framework > > provides an infrastructure in which "classes" can per- > > form transformations on disk I/O requests on their path > > from the upper kernel to the device drivers and back. > > > > Could it be, that geom slows something down (in some boxes the > > reading > > ops are very slow; in my box the writing ops are very slow)? > > [...] > against a further offset region of ad0. If they're against the > same bits of disk, the main difference here will be the > additional processing of the layers in the stack. A little > bit of math is required to figure out the offset, but dd > should be usable to figure out the incremental cost. > I used the following commands: 1. dd if=/dev/ad0s2a bs=16k count=10000 of=/dev/null 2. dd if=/dev/ad0 iseek=2100357 bs=16k count=10000 of=/dev/null (I think I did the math correctly; indeed the read speed of my ad0 varies between 45MB/sec (iseek=0) and 25MB/sec (in the end) with bs=128k). The results were nearly the same (both between 26MB/sec and 28MB/sec). Maybe I should have done it in single user mode. My other hard disc ad1 (it is newer and bigger and faster and more furious) behaves better (but I can just try writing via the file system (ufs+s) and a final sync): The sync just needs .24sec of 18.28 seconds; the read and write speed is nearly the same (about 37MB/sec+/-1MB/sec). Is there anything else, that could help to find the reason for the difference between ad0's speed in R4.11 and R5.3? -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Sun May 1 13:46:39 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 150F716A4CE for ; Sun, 1 May 2005 13:46:39 +0000 (GMT) Received: from avscan1.sentex.ca (avscan1.sentex.ca [199.212.134.11]) by mx1.FreeBSD.org (Postfix) with ESMTP id A005843D2F for ; Sun, 1 May 2005 13:46:38 +0000 (GMT) (envelope-from mike@sentex.net) Received: from localhost (localhost.sentex.ca [127.0.0.1]) by avscan1.sentex.ca (8.12.11/8.12.11) with ESMTP id j41DkbdW041581 for ; Sun, 1 May 2005 09:46:38 -0400 (EDT) (envelope-from mike@sentex.net) Received: from avscan1.sentex.ca ([127.0.0.1]) by localhost (avscan1.sentex.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 40770-09 for ; Sun, 1 May 2005 09:46:37 -0400 (EDT) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by avscan1.sentex.ca (8.12.11/8.12.11) with ESMTP id j41Dkb3R041568 for ; Sun, 1 May 2005 09:46:37 -0400 (EDT) (envelope-from mike@sentex.net) Received: from simian.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.3/8.12.11) with ESMTP id j41DkVr5089761 for ; Sun, 1 May 2005 09:46:31 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Sun, 01 May 2005 09:46:09 -0400 To: freebsd-performance@freebsd.org From: Mike Tancsa Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: by amavisd-new X-Virus-Scanned: by amavisd-new at avscan1b Subject: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2005 13:46:39 -0000 A somewhat obvious question to some perhaps, but what server application mix on FreeBSD today sees an improvement using 64bit CPUs ? In my ISP centric world, my big apps are BIND, IMAP/POP3, httpd via apache, SMTP, AV and SPAM scanning, and firewalls/routing. Apart from larger RAM, why would these benefit from the 64bit world ? Or would they ? ---Mike -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike From owner-freebsd-performance@FreeBSD.ORG Sun May 1 16:49:14 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DD06C16A4CE for ; Sun, 1 May 2005 16:49:14 +0000 (GMT) Received: from pi.codefab.com (pi.codefab.com [199.103.21.227]) by mx1.FreeBSD.org (Postfix) with ESMTP id A4E5A43D31 for ; Sun, 1 May 2005 16:49:14 +0000 (GMT) (envelope-from cswiger@mac.com) Received: from localhost (localhost [127.0.0.1]) by pi.codefab.com (Postfix) with ESMTP id 186585E8D; Sun, 1 May 2005 12:49:14 -0400 (EDT) Received: from pi.codefab.com ([127.0.0.1]) by localhost (pi.codefab.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 31990-07; Sun, 1 May 2005 12:49:13 -0400 (EDT) Received: from [192.168.1.3] (pool-68-161-53-96.ny325.east.verizon.net [68.161.53.96]) by pi.codefab.com (Postfix) with ESMTP id 041765C86; Sun, 1 May 2005 12:49:12 -0400 (EDT) Message-ID: <42750861.8000509@mac.com> Date: Sun, 01 May 2005 12:48:33 -0400 From: Chuck Swiger Organization: The Courts of Chaos User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.7) Gecko/20050414 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mike Tancsa References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> In-Reply-To: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: amavisd-new at codefab.com cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2005 16:49:15 -0000 Mike Tancsa wrote: > A somewhat obvious question to some perhaps, but what server application > mix on FreeBSD today sees an improvement using 64bit CPUs ? Databases. Big ones, anyway. Other than that, not much, unless you're running processes which would like to use more than 2GB of RAM. > In my ISP centric world, my big apps are BIND, IMAP/POP3, httpd via apache, > SMTP, AV and SPAM scanning, and firewalls/routing. Apart from larger RAM, > why would these benefit from the 64bit world ? Or would they ? None of these tasks would benefit much from 64-bit computing; many of them might even run faster in 32-bit mode than in 64-bit mode. -- -Chuck From owner-freebsd-performance@FreeBSD.ORG Sun May 1 19:10:03 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D2DA216A4CE for ; Sun, 1 May 2005 19:10:03 +0000 (GMT) Received: from mortis.over-yonder.net (adsl-12-42-198.jan.bellsouth.net [65.12.42.198]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0829443D45 for ; Sun, 1 May 2005 19:10:03 +0000 (GMT) (envelope-from fullermd@over-yonder.net) Received: by mortis.over-yonder.net (Postfix, from userid 100) id ACAB320F5A; Sun, 1 May 2005 14:10:01 -0500 (CDT) Date: Sun, 1 May 2005 14:10:01 -0500 From: "Matthew D. Fuller" To: Mike Tancsa Message-ID: <20050501191001.GF85317@over-yonder.net> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.9i-fullermd.2 cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2005 19:10:03 -0000 On Sun, May 01, 2005 at 09:46:09AM -0400 I heard the voice of Mike Tancsa, and lo! it spake thus: > > A somewhat obvious question to some perhaps, but what server > application mix on FreeBSD today sees an improvement using 64bit > CPUs ? It should be noted that according to most benchmarks I've seen, the amd64 chips are the fastest 32-bit chips available. So, even if you're not using the full 64-bit capability in your apps (or even just running them purely in 32-bit mode), you're in the green. There are other advantage to them (like register space) than simply the 32/64 split. And don't forget bragging rights! Servers always run faster when they're cool... -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-performance@FreeBSD.ORG Sun May 1 23:34:59 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E1E7D16A4CE for ; Sun, 1 May 2005 23:34:59 +0000 (GMT) Received: from avscan1.sentex.ca (avscan1.sentex.ca [199.212.134.11]) by mx1.FreeBSD.org (Postfix) with ESMTP id 81FE643D4C for ; Sun, 1 May 2005 23:34:59 +0000 (GMT) (envelope-from mike@sentex.net) Received: from localhost (localhost.sentex.ca [127.0.0.1]) by avscan1.sentex.ca (8.12.11/8.12.11) with ESMTP id j41NYwtQ022343; Sun, 1 May 2005 19:34:58 -0400 (EDT) (envelope-from mike@sentex.net) Received: from avscan1.sentex.ca ([127.0.0.1]) by localhost (avscan1.sentex.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 22118-03; Sun, 1 May 2005 19:34:58 -0400 (EDT) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by avscan1.sentex.ca (8.12.11/8.12.11) with ESMTP id j41NYvO4022321; Sun, 1 May 2005 19:34:57 -0400 (EDT) (envelope-from mike@sentex.net) Received: from simian.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.3/8.12.11) with ESMTP id j41NYphR090751; Sun, 1 May 2005 19:34:52 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <6.2.1.2.0.20050501183331.045c3770@64.7.153.2> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Sun, 01 May 2005 19:34:31 -0400 To: "Matthew D. Fuller" From: Mike Tancsa In-Reply-To: <20050501191001.GF85317@over-yonder.net> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <20050501191001.GF85317@over-yonder.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: by amavisd-new X-Virus-Scanned: by amavisd-new at avscan1b cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2005 23:35:00 -0000 At 03:10 PM 01/05/2005, Matthew D. Fuller wrote: >On Sun, May 01, 2005 at 09:46:09AM -0400 I heard the voice of >Mike Tancsa, and lo! it spake thus: > > > > A somewhat obvious question to some perhaps, but what server > > application mix on FreeBSD today sees an improvement using 64bit > > CPUs ? > >It should be noted that according to most benchmarks I've seen, the >amd64 chips are the fastest 32-bit chips available. I havent paid that much attention to the comparisons as they tend to be very windows based and games to boot. But surely its hard to compare across platforms. The CPUs require different chipsets, so some of the performance results can be due to the MB and RAM. http://www6.tomshardware.com/cpu/20050221/prescott-10.html#synthetic The P4 640 3.2Ghz is about the same price at one of my suppliers as the 64 3500+ and the numbers are not that different. For me to switch to a different platform and risk stability issues there would have to be some decent payoff in either cost savings or serious performance differences. I am thinking perhaps for my RADIUS accounting database which is very large might benefit as I have already maxed out the RAM. >And don't forget bragging rights! Servers always run faster when >they're cool... Nahhh... I am an uncool kind of guy. Hell, my glasses are still from the 80s! ---Mike From owner-freebsd-performance@FreeBSD.ORG Sun May 1 23:42:19 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0BE5F16A4CE for ; Sun, 1 May 2005 23:42:19 +0000 (GMT) Received: from avscan1.sentex.ca (avscan1.sentex.ca [199.212.134.11]) by mx1.FreeBSD.org (Postfix) with ESMTP id A8C1043D1F for ; Sun, 1 May 2005 23:42:18 +0000 (GMT) (envelope-from mike@sentex.net) Received: from localhost (localhost.sentex.ca [127.0.0.1]) by avscan1.sentex.ca (8.12.11/8.12.11) with ESMTP id j41NgH8j023128; Sun, 1 May 2005 19:42:17 -0400 (EDT) (envelope-from mike@sentex.net) Received: from avscan1.sentex.ca ([127.0.0.1]) by localhost (avscan1.sentex.ca [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 22818-04; Sun, 1 May 2005 19:42:17 -0400 (EDT) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by avscan1.sentex.ca (8.12.11/8.12.11) with ESMTP id j41NgHBK023105; Sun, 1 May 2005 19:42:17 -0400 (EDT) (envelope-from mike@sentex.net) Received: from simian.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.3/8.12.11) with ESMTP id j41NgBDm090774; Sun, 1 May 2005 19:42:11 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <6.2.1.2.0.20050501193659.05483bd0@64.7.153.2> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Sun, 01 May 2005 19:41:52 -0400 To: Chuck Swiger From: Mike Tancsa In-Reply-To: <42750861.8000509@mac.com> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <42750861.8000509@mac.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: by amavisd-new X-Virus-Scanned: by amavisd-new at avscan1b cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 May 2005 23:42:19 -0000 At 12:48 PM 01/05/2005, Chuck Swiger wrote: >Mike Tancsa wrote: >>A somewhat obvious question to some perhaps, but what server application >>mix on FreeBSD today sees an improvement using 64bit CPUs ? > >Databases. Big ones, anyway. Other than that, not much, unless you're >running processes which would like to use more than 2GB of RAM. Thanks! I was worried I was missing something obvious in all the 64bit excitement. For my apps where I cant balance across multiple machines or where space is an issue, dual core CPUs I think might be more interesting to look at in a few months. ---Mike From owner-freebsd-performance@FreeBSD.ORG Mon May 2 00:07:58 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7182016A4CE for ; Mon, 2 May 2005 00:07:58 +0000 (GMT) Received: from outgoing.redshift.com (outgoing.redshift.com [207.177.231.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id 308EF43D48 for ; Mon, 2 May 2005 00:07:58 +0000 (GMT) (envelope-from ray@redshift.com) Received: from workstation (216-228-19-21.dsl.redshift.com [216.228.19.21]) by outgoing.redshift.com (Postfix) with SMTP id 34F3E97090; Sun, 1 May 2005 17:07:52 -0700 (PDT) Message-Id: <3.0.1.32.20050501170802.00b10c58@pop.redshift.com> X-Mailer: na X-Sender: redshift.com Date: Sun, 01 May 2005 17:08:02 -0700 To: Chuck Swiger , Mike Tancsa From: ray@redshift.com In-Reply-To: <42750861.8000509@mac.com> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <6.2.1.2.0.20050501094429.06974910@64.7.153.2> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 00:07:58 -0000 my benchmarking with apache shows that it does seem to run faster on 32 bit Xeon than 64 bit Opterons. PHP ran a bit faster on the Opterons, but nothing major. Ray At 12:48 PM 5/1/2005 -0400, Chuck Swiger wrote: | Mike Tancsa wrote: | > A somewhat obvious question to some perhaps, but what server application | > mix on FreeBSD today sees an improvement using 64bit CPUs ? | | Databases. Big ones, anyway. Other than that, not much, unless you're | running processes which would like to use more than 2GB of RAM. | | > In my ISP centric world, my big apps are BIND, IMAP/POP3, httpd via apache, | > SMTP, AV and SPAM scanning, and firewalls/routing. Apart from larger RAM, | > why would these benefit from the 64bit world ? Or would they ? | | None of these tasks would benefit much from 64-bit computing; many of them | might even run faster in 32-bit mode than in 64-bit mode. | | -- | -Chuck | _______________________________________________ | freebsd-performance@freebsd.org mailing list | http://lists.freebsd.org/mailman/listinfo/freebsd-performance | To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" | | From owner-freebsd-performance@FreeBSD.ORG Mon May 2 03:45:09 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2CF7F16A4CE for ; Mon, 2 May 2005 03:45:09 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id C29D643D1F for ; Mon, 2 May 2005 03:45:08 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 17E2246B8E; Sun, 1 May 2005 23:45:08 -0400 (EDT) Date: Mon, 2 May 2005 04:48:12 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Mike Tancsa In-Reply-To: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> Message-ID: <20050502044655.E78953@fledge.watson.org> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 03:45:09 -0000 On Sun, 1 May 2005, Mike Tancsa wrote: > A somewhat obvious question to some perhaps, but what server application > mix on FreeBSD today sees an improvement using 64bit CPUs ? In my ISP > centric world, my big apps are BIND, IMAP/POP3, httpd via apache, SMTP, > AV and SPAM scanning, and firewalls/routing. Apart from larger RAM, why > would these benefit from the 64bit world ? Or would they ? RAM/address space is the big reason. In fact, applications compiled for 64-bits may well run slower than 32-bit ones running on the 64-bit kernel. Note that systems like Solaris default build many programs as 32-bit, since there's no benefit to running ls(1) with a 64-bit address space. On the other hand, if you need a large memory system, running a 64-bit system will give you better support and utilization than a 32-bit system runing with PAE, at least on FreeBSD. Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Mon May 2 04:16:22 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 42DCD16A4CF for ; Mon, 2 May 2005 04:16:22 +0000 (GMT) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.202]) by mx1.FreeBSD.org (Postfix) with ESMTP id CA95043D49 for ; Mon, 2 May 2005 04:16:21 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: by rproxy.gmail.com with SMTP id j1so828764rnf for ; Sun, 01 May 2005 21:16:21 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=onjEdFKTJ5jEQzivL6iA8xHb0Evoqs2AvpcqF5jsdtOVt6YcQU7vKBvdZP9481z68ys0u9sanEXFmZBrVuZa6gXFNQwEZnLLpNyWm9z7H1Ke4ybD+2+1KQm1oroTYLcFu6JkDwFfJk+A323hLEbtM0kMfLoAmo5j8omUd7pVVNQ= Received: by 10.38.11.53 with SMTP id 53mr5060883rnk; Sun, 01 May 2005 21:16:21 -0700 (PDT) Received: by 10.38.209.22 with HTTP; Sun, 1 May 2005 21:16:21 -0700 (PDT) Message-ID: <84dead7205050121166c326294@mail.gmail.com> Date: Mon, 2 May 2005 04:16:21 +0000 From: Joseph Koshy To: Robert Watson In-Reply-To: <20050502044655.E78953@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <20050502044655.E78953@fledge.watson.org> cc: freebsd-performance@freebsd.org cc: Mike Tancsa Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Joseph Koshy List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 04:16:22 -0000 > RAM/address space is the big reason. In fact, applications=20 > compiled for 64-bits may well run slower than 32-bit ones=20 > running on the 64-bit kernel. On the other hand we have 16 registers to play with on the AMD64 and they can be used far more orthogonally than on the i386. That would cut down on the frequent register spills and reloads that are the bane of the i386 and thus help improve performance. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 05:17:15 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4907316A4CE; Mon, 2 May 2005 05:17:15 +0000 (GMT) Received: from coe.ufrj.br (roma.coe.ufrj.br [146.164.53.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 91EB643D41; Mon, 2 May 2005 05:17:14 +0000 (GMT) (envelope-from jonny@jonny.eng.br) Received: from localhost (localhost [127.0.0.1]) by coe.ufrj.br (Postfix) with ESMTP id C9E8017017; Mon, 2 May 2005 02:17:10 -0300 (BRT) Received: from coe.ufrj.br ([146.164.53.65]) by localhost (roma.coe.ufrj.br [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 09909-08; Mon, 2 May 2005 02:17:07 -0300 (BRT) Received: from [200.164.156.123] (unknown [200.164.156.123]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by coe.ufrj.br (Postfix) with ESMTP id 39F4517015; Mon, 2 May 2005 02:17:06 -0300 (BRT) Message-ID: <4275B7CA.9040701@jonny.eng.br> Date: Mon, 02 May 2005 02:16:58 -0300 From: =?ISO-8859-1?Q?Jo=E3o_Carlos_Mendes_Lu=EDs?= User-Agent: Mozilla Thunderbird 1.0+ (Windows/20050329) MIME-Version: 1.0 To: Robert Watson References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <20050502044655.E78953@fledge.watson.org> In-Reply-To: <20050502044655.E78953@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: amavisd-new at coe.ufrj.br cc: freebsd-performance@freebsd.org cc: Mike Tancsa Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 05:17:15 -0000 Robert Watson wrote: > On Sun, 1 May 2005, Mike Tancsa wrote: >> A somewhat obvious question to some perhaps, but what server=20 >> application mix on FreeBSD today sees an improvement using 64bit CPUs = >> ? In my ISP centric world, my big apps are BIND, IMAP/POP3, httpd via= =20 >> apache, SMTP, AV and SPAM scanning, and firewalls/routing. Apart from= =20 >> larger RAM, why would these benefit from the 64bit world ? Or would=20 >> they ? >=20 > RAM/address space is the big reason. In fact, applications compiled fo= r=20 > 64-bits may well run slower than 32-bit ones running on the 64-bit=20 > kernel. Note that systems like Solaris default build many programs as=20 > 32-bit, since there's no benefit to running ls(1) with a 64-bit address= =20 > space. Should I understand from this that, in the long term, FreeBSD will=20 take the same path as Solaris, and have a dual (32bit/64bit) userland on = amd64 arch? Jonny --=20 Jo=E3o Carlos Mendes Lu=EDs - Networking Engineer - jonny@jonny.eng.br From owner-freebsd-performance@FreeBSD.ORG Mon May 2 12:16:35 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 41AF916A4D0 for ; Mon, 2 May 2005 12:16:35 +0000 (GMT) Received: from wask.wask.wroc.pl (wask.wask.wroc.pl [156.17.254.3]) by mx1.FreeBSD.org (Postfix) with ESMTP id D687943D46 for ; Mon, 2 May 2005 12:16:34 +0000 (GMT) (envelope-from k0re@wask.wask.wroc.pl) Received: by wask.wask.wroc.pl (Postfix, from userid 143) id B5FBA7CFA1; Mon, 2 May 2005 14:16:33 +0200 (MEST) Date: Mon, 2 May 2005 14:16:33 +0200 From: Mariusz Grad To: Mike Tancsa Message-ID: <20050502121633.GA14896@wask.wask.wroc.pl> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> X-Organization: [ WCSS ] X-Mailman-Approved-At: Mon, 02 May 2005 12:38:33 +0000 cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 12:16:35 -0000 Mike Tancsa: > A somewhat obvious question to some perhaps, but what server application > mix on FreeBSD today sees an improvement using 64bit CPUs ? In my ISP > centric world, my big apps are BIND, IMAP/POP3, httpd via apache, SMTP, AV > and SPAM scanning, and firewalls/routing. Apart from larger RAM, why would > these benefit from the 64bit world ? Or would they ? Benefits from AMD64: - larger RAM limit (40bit memory address), - 64bit GRPs, - much faster access to memory (memory controller inside core - no northbridge), - it much better scales (with many CPUs) (he doesnt share memory bandwith), - large caches L1 = 64KB, L2 = 1MB (comparing to xeon 12/512K), - DEP = Data Execution Protection (it has to be supported via OS), - Multimedia Extensions for graphics / vector processing. = Opterons remove memory bottlenecks espacially with multi-cpus. Ive had large acceleration at: - math computation (f77, 2-4GB double precision matrix) single opteron 1.8GHz (sun v20z) was 4x faster then celeron 2.4GHz. - databases (pgsql) Opteron 1.8 GHz 2x faster then Xeon 3.2 I would assume that everything which can run: - in parallel, - on multi-Opterons CPU, - which uses heavly RAM, - which can take benefits from 64bit registers (floating points), will have _extreamly_ boost comparing to Intels EMT64 (or Itanium2). -- Mariusz Grad / UNIX Systems Administrator tel: +4871 320-2520 mobile: +48 501-134-008 Wroclaw Centre for Networking and Supercomputing address: Wybrzeze Wyspianskiego 27, Wroclaw, Poland From owner-freebsd-performance@FreeBSD.ORG Mon May 2 12:41:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1B45C16A4CF; Mon, 2 May 2005 12:41:10 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 01CED43D31; Mon, 2 May 2005 12:41:09 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j42Cf7SS004598; Mon, 2 May 2005 07:41:08 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42761FA7.1030500@centtech.com> Date: Mon, 02 May 2005 07:40:07 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: =?ISO-8859-15?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= References: <20050501112351.4184.qmail@web41205.mail.yahoo.com> In-Reply-To: <20050501112351.4184.qmail@web41205.mail.yahoo.com> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: ClamAV 0.82/861/Sat Apr 30 04:28:52 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org cc: Robert Watson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 12:41:10 -0000 Arne W=F6rner wrote: > --- Robert Watson wrote: >=20 >>On Sat, 30 Apr 2005, Arne WXrner wrote: >> >>>3. The man page geom(4) of R5.3 says "The GEOM framework >>> provides an infrastructure in which "classes" can per- >>> form transformations on disk I/O requests on their path >>> from the upper kernel to the device drivers and back. >>> >>>Could it be, that geom slows something down (in some boxes the >>>reading=20 >>>ops are very slow; in my box the writing ops are very slow)? >> >>[...] >>against a further offset region of ad0. If they're against the >>same bits of disk, the main difference here will be the >>additional processing of the layers in the stack. A little >>bit of math is required to figure out the offset, but dd >>should be usable to figure out the incremental cost. >> >=20 > I used the following commands: > 1. dd if=3D/dev/ad0s2a bs=3D16k count=3D10000 of=3D/dev/null > 2. dd if=3D/dev/ad0 iseek=3D2100357 bs=3D16k count=3D10000 of=3D/dev/nu= ll > (I think I did the math correctly; indeed the read speed of my ad0 > varies between 45MB/sec (iseek=3D0) and 25MB/sec (in the end) with > bs=3D128k). > The results were nearly the same (both between 26MB/sec and > 28MB/sec). Maybe I should have done it in single user mode. >=20 > My other hard disc ad1 (it is newer and bigger and faster and more > furious) behaves better (but I can just try writing via the file > system (ufs+s) and a final sync): The sync just needs .24sec of > 18.28 seconds; the read and write speed is nearly the same (about > 37MB/sec+/-1MB/sec). >=20 > Is there anything else, that could help to find the reason for the > difference between ad0's speed in R4.11 and R5.3? I'll be honest here, I don't care much if the speed difference between=20 4.X and 5.X is measureable, or whatever. What I find is a little=20 telling of an issue somewhere, is that READS are slower than WRITES!=20 This is totally bogus to me - dd'ing a file to a filesystem, then=20 umounting should take longer than dd'ing from the disk to /dev/null, it=20 nearly every config I can dream up. Maybe it's the speed at which=20 /dev/null can gobble bits (seems highly unlikely!), or maybe GEOM is=20 busy doing a check or some routine to data being accessed directly from=20 the disk device instead of through a filesystem? I don't know, but it=20 is an issue, and I'm sure we'll get nailed up to a fence in some=20 benchmark somewhere if we don't fix it.. Eric --=20 ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon May 2 12:44:48 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E330A16A4CE; Mon, 2 May 2005 12:44:48 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 33E2D43D54; Mon, 2 May 2005 12:44:48 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id E02EA46B37; Mon, 2 May 2005 08:44:47 -0400 (EDT) Date: Mon, 2 May 2005 13:47:55 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: performance@FreeBSD.org Message-ID: <20050502134725.G87351@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: scottl@FreeBSD.org Subject: Malloc statistics patch X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 12:44:49 -0000 Now that the UMA changes to use critical sections instead of per-cpu mutexes for per-cpu caches are merged, I'd like to move on to the next piece of the memory allocation patch previous posted. The attached patch does a couple of things: - Introduce per-cpu malloc type statistics, protecting them with critical sections. This replaces previous use of a mutex per type to protect cross-cpu allocation information. - Implement coalescing of per-cpu malloc type statistics when reporting for kern.malloc, used by vmstat -m, and when inspecting a core dump/kernel memory directly. - Maintain kernel ABI compatibility for MALLOC_DECLARE(), so that if it's desirable to merge these changes to 5.x, we can do so without breaking third-party kernel modles. Ideally these chances should make the system faster, substiting the use of per-CPU memory for statistics, replacing a mutex with critical section use, etc. I've confirmed it has positive effects using micro-benchmarks, but would appreciate assistance in evaluating the impact on macro-benchmarks. The change is likely small, but should be measurable, especially on CPU-bound loads. Stability testing under load would also go much appreciated. Notes: - The malloc type peak 'HighUse' field is no longer easily maintained, since there's no longer a global notion of how much memory is allocated for each maloc type. As such, continuous maintenance is no longer possible. We could replace this with polled or unsynchronized maintenance of a High Use value. - If we're willing to abandon 5.x backport, we can clean up 'struct malloc_type' quite a bit, while maintaining memory ownership of malloc type storage in the malloc code to avoid future ABI breakage. We might wish to use the compatibility code in 6.x for now, merge to 5.x, then remove from 6.x. Robert N M Watson --- //depot/vendor/freebsd/src/usr.bin/vmstat/vmstat.c 2005/02/21 14:38:18 +++ //depot/user/rwatson/percpu/src/usr.bin/vmstat/vmstat.c 2005/04/30 10:39:05 @@ -46,6 +46,8 @@ #include __FBSDID("$FreeBSD: src/usr.bin/vmstat/vmstat.c,v 1.84 2005/02/21 14:35:00 ps Exp $"); +#define WANT_MALLOC_TYPE_INTERNAL + #include #include #include @@ -897,10 +899,22 @@ (long long)inttotal, (long long)(inttotal / uptime)); } +/* + * domem() replicates the kernel implementation of kern.malloc by inspecting + * kernel data structures, which is appropriate for use on a core dump. + * domem() must identify the set of malloc types, walk the list, and coalesce + * per-CPU state to report. It relies on direct access to internal kernel + * data structures that have a fragile (and intentionally unexposed) ABI. + * This logic should not be used by live monitoring tools, which should + * instead rely solely on the sysctl interface. + */ static void domem(void) { + struct malloc_type_stats *mts, mts_local; + struct malloc_type_internal mti; struct malloc_type type; + int i; if (kd == NULL) { dosysctl("kern.malloc"); @@ -924,15 +938,34 @@ sizeof(type)) errx(1, "%s: %p: %s", __func__, type.ks_next, kvm_geterr(kd)); - if (type.ks_calls == 0) + if (kvm_read(kd, (u_long)type.ks_handle, &mti, sizeof(mti)) != + sizeof(mti)) + errx(1, "%s: %p: %s", __func__, type.ks_handle, + kvm_geterr(kd)); + + bzero(&mts_local, sizeof(mts_local)); + for (i = 0; i < MAXCPU; i++) { + mts = &mti.mti_stats[i]; + mts_local.mts_memalloced += mts->mts_memalloced; + mts_local.mts_memfreed += mts->mts_memfreed; + mts_local.mts_numallocs += mts->mts_numallocs; + mts_local.mts_numfrees += mts->mts_numfrees; + mts_local.mts_size |= mts->mts_size; + } + if (mts_local.mts_numallocs == 0) continue; + + /* + * Unlike in kern_malloc.c, we don't mask inter-CPU races, as * vmstat on a core is likely for debugging purposes. + */ + str = kgetstr(type.ks_shortdesc); - (void)printf("%13s%6lu%6luK%7luK%9llu", + (void)printf("%13s%6lu%6luK -%9llu", str, - type.ks_inuse, - (type.ks_memuse + 1023) / 1024, - (type.ks_maxused + 1023) / 1024, - (long long unsigned)type.ks_calls); + mts_local.mts_numallocs - mts_local.mts_numfrees, + ((mts_local.mts_memalloced - mts_local.mts_memfreed) + + 1023) / 1024, + mts_local.mts_numallocs); free(str); for (kmemzonenum = 0, first = 1; ; kmemzonenum++) { kreado(X_KMEMZONES, &kz, sizeof(kz), @@ -941,7 +974,7 @@ (void)printf("\n"); break; } - if (!(type.ks_size & (1 << kmemzonenum))) + if (!(mts_local.mts_size & (1 << kmemzonenum))) continue; if (first) (void)printf(" "); --- //depot/vendor/freebsd/src/sys/kern/kern_malloc.c 2005/04/12 23:55:38 +++ //depot/user/rwatson/percpu/sys/kern/kern_malloc.c 2005/04/30 14:22:11 @@ -1,4 +1,5 @@ /*- + * Copyright (c) 2005 Robert N. M. Watson * Copyright (c) 1987, 1991, 1993 * The Regents of the University of California. All rights reserved. * @@ -34,6 +35,8 @@ #include "opt_vm.h" +#define WANT_MALLOC_TYPE_INTERNAL + #include #include #include @@ -44,6 +47,7 @@ #include #include #include +#include #include #include @@ -133,6 +137,8 @@ {0, NULL}, }; +uma_zone_t mt_zone; + #ifdef DEBUG_MEMGUARD u_int vm_memguard_divisor; SYSCTL_UINT(_vm, OID_AUTO, memguard_divisor, CTLFLAG_RD, &vm_memguard_divisor, @@ -197,41 +203,44 @@ * Add this to the informational malloc_type bucket. */ static void -malloc_type_zone_allocated(struct malloc_type *ksp, unsigned long size, +malloc_type_zone_allocated(struct malloc_type *type, unsigned long size, int zindx) { - mtx_lock(&ksp->ks_mtx); - ksp->ks_calls++; + struct malloc_type_internal *mti; + struct malloc_type_stats *mts; + + critical_enter(); + mti = (struct malloc_type_internal *)(type->ks_handle); + mts = &mti->mti_stats[curcpu]; + mts->mts_memalloced += size; + mts->mts_numallocs++; if (zindx != -1) - ksp->ks_size |= 1 << zindx; - if (size != 0) { - ksp->ks_memuse += size; - ksp->ks_inuse++; - if (ksp->ks_memuse > ksp->ks_maxused) - ksp->ks_maxused = ksp->ks_memuse; - } - mtx_unlock(&ksp->ks_mtx); + mts->mts_size |= 1 << zindx; + critical_exit(); } void -malloc_type_allocated(struct malloc_type *ksp, unsigned long size) +malloc_type_allocated(struct malloc_type *type, unsigned long size) { - malloc_type_zone_allocated(ksp, size, -1); + + malloc_type_zone_allocated(type, size, -1); } /* * Remove this allocation from the informational malloc_type bucket. */ void -malloc_type_freed(struct malloc_type *ksp, unsigned long size) +malloc_type_freed(struct malloc_type *type, unsigned long size) { - mtx_lock(&ksp->ks_mtx); - KASSERT(size <= ksp->ks_memuse, - ("malloc(9)/free(9) confusion.\n%s", - "Probably freeing with wrong type, but maybe not here.")); - ksp->ks_memuse -= size; - ksp->ks_inuse--; - mtx_unlock(&ksp->ks_mtx); + struct malloc_type_internal *mti; + struct malloc_type_stats *mts; + + critical_enter(); + mti = (struct malloc_type_internal *)type->ks_handle; + mts = &mti->mti_stats[curcpu]; + mts->mts_memfreed += size; + mts->mts_numfrees++; + critical_exit(); } /* @@ -351,9 +360,6 @@ } #endif - KASSERT(type->ks_memuse > 0, - ("malloc(9)/free(9) confusion.\n%s", - "Probably freeing with wrong type, but maybe not here.")); size = 0; slab = vtoslab((vm_offset_t)addr & (~UMA_SLAB_MASK)); @@ -405,6 +411,11 @@ if (addr == NULL) return (malloc(size, type, flags)); + /* + * XXX: Should report free of old memory and alloc of new memory to + * per-CPU stats. + */ + #ifdef DEBUG_MEMGUARD /* XXX: CHANGEME! */ if (type == M_SUBPROC) { @@ -543,6 +554,13 @@ uma_startup2(); + mt_zone = uma_zcreate("mt_zone", sizeof(struct malloc_type_internal), +#ifdef INVARIANTS + mtrash_ctor, mtrash_dtor, mtrash_init, mtrash_fini, +#else + NULL, NULL, NULL, NULL, +#endif + UMA_ALIGN_PTR, UMA_ZONE_MALLOC); for (i = 0, indx = 0; kmemzones[indx].kz_size != 0; indx++) { int size = kmemzones[indx].kz_size; char *name = kmemzones[indx].kz_name; @@ -562,127 +580,143 @@ } void -malloc_init(void *data) +malloc_init(void *type) { - struct malloc_type *type = (struct malloc_type *)data; + struct malloc_type_internal *mti; + struct malloc_type *mt; - mtx_lock(&malloc_mtx); - if (type->ks_magic != M_MAGIC) - panic("malloc type lacks magic"); + KASSERT(cnt.v_page_count != 0, ("malloc_register before vm_init")); - if (cnt.v_page_count == 0) - panic("malloc_init not allowed before vm init"); + mt = type; + mti = uma_zalloc(mt_zone, M_WAITOK | M_ZERO); + mt->ks_handle = mti; - if (type->ks_next != NULL) - return; - - type->ks_next = kmemstatistics; + mtx_lock(&malloc_mtx); + mt->ks_next = kmemstatistics; kmemstatistics = type; - mtx_init(&type->ks_mtx, type->ks_shortdesc, "Malloc Stats", MTX_DEF); mtx_unlock(&malloc_mtx); } void -malloc_uninit(void *data) +malloc_uninit(void *type) { - struct malloc_type *type = (struct malloc_type *)data; - struct malloc_type *t; + struct malloc_type_internal *mti; + struct malloc_type *mt, *temp; + mt = type; + KASSERT(mt->ks_handle != NULL, ("malloc_deregister: cookie NULL")); mtx_lock(&malloc_mtx); - mtx_lock(&type->ks_mtx); - if (type->ks_magic != M_MAGIC) - panic("malloc type lacks magic"); - - if (cnt.v_page_count == 0) - panic("malloc_uninit not allowed before vm init"); - - if (type == kmemstatistics) - kmemstatistics = type->ks_next; - else { - for (t = kmemstatistics; t->ks_next != NULL; t = t->ks_next) { - if (t->ks_next == type) { - t->ks_next = type->ks_next; - break; - } + mti = mt->ks_handle; + mt->ks_handle = NULL; + if (mt != kmemstatistics) { + for (temp = kmemstatistics; temp != NULL; + temp = temp->ks_next) { + if (temp->ks_next == mt) + temp->ks_next = mt->ks_next; } - } - type->ks_next = NULL; - mtx_destroy(&type->ks_mtx); + } else + kmemstatistics = mt->ks_next; mtx_unlock(&malloc_mtx); + uma_zfree(mt_zone, type); } static int sysctl_kern_malloc(SYSCTL_HANDLER_ARGS) { + struct malloc_type_stats *mts, mts_local; + struct malloc_type_internal *mti; + long temp_allocs, temp_bytes; struct malloc_type *type; int linesize = 128; - int curline; + struct sbuf sbuf; int bufsize; int first; int error; char *buf; - char *p; int cnt; - int len; int i; cnt = 0; + /* Guess at how much room is needed. */ mtx_lock(&malloc_mtx); for (type = kmemstatistics; type != NULL; type = type->ks_next) cnt++; + mtx_unlock(&malloc_mtx); - mtx_unlock(&malloc_mtx); bufsize = linesize * (cnt + 1); - p = buf = (char *)malloc(bufsize, M_TEMP, M_WAITOK|M_ZERO); + buf = (char *)malloc(bufsize, M_TEMP, M_WAITOK|M_ZERO); + sbuf_new(&sbuf, buf, bufsize, SBUF_FIXEDLEN); + mtx_lock(&malloc_mtx); - - len = snprintf(p, linesize, + sbuf_printf(&sbuf, "\n Type InUse MemUse HighUse Requests Size(s)\n"); - p += len; - for (type = kmemstatistics; cnt != 0 && type != NULL; type = type->ks_next, cnt--) { - if (type->ks_calls == 0) + mti = type->ks_handle; + bzero(&mts_local, sizeof(mts_local)); + for (i = 0; i < MAXCPU; i++) { + mts = &mti->mti_stats[i]; + mts_local.mts_memalloced += mts->mts_memalloced; + mts_local.mts_memfreed += mts->mts_memfreed; + mts_local.mts_numallocs += mts->mts_numallocs; + mts_local.mts_numfrees += mts->mts_numfrees; + mts_local.mts_size |= mts->mts_size; + } + if (mts_local.mts_numallocs == 0) continue; - curline = linesize - 2; /* Leave room for the \n */ - len = snprintf(p, curline, "%13s%6lu%6luK%7luK%9llu", - type->ks_shortdesc, - type->ks_inuse, - (type->ks_memuse + 1023) / 1024, - (type->ks_maxused + 1023) / 1024, - (long long unsigned)type->ks_calls); - curline -= len; - p += len; + /* + * Due to races in per-CPU statistics gather, it's possible to + * get a slightly negative number here. If we do, approximate + * with 0. + */ + if (mts_local.mts_numallocs > mts_local.mts_numfrees) + temp_allocs = mts_local.mts_numallocs - + mts_local.mts_numfrees; + else + temp_allocs = 0; + + /* + * Ditto for bytes allocated. + */ + if (mts_local.mts_memalloced > mts_local.mts_memfreed) + temp_bytes = mts_local.mts_memalloced - + mts_local.mts_memfreed; + else + temp_bytes = 0; + + /* + * XXXRW: High-waterwark is no longer easily available, so + * we just print '-' for that column. + */ + sbuf_printf(&sbuf, "%13s%6lu%6luK -%9lu", + type->ks_shortdesc, + temp_allocs, + (temp_bytes + 1023) / 1024, + mts_local.mts_numallocs); first = 1; for (i = 0; i < sizeof(kmemzones) / sizeof(kmemzones[0]) - 1; i++) { - if (type->ks_size & (1 << i)) { + if (mts_local.mts_size & (1 << i)) { if (first) - len = snprintf(p, curline, " "); + sbuf_printf(&sbuf, " "); else - len = snprintf(p, curline, ","); - curline -= len; - p += len; - - len = snprintf(p, curline, - "%s", kmemzones[i].kz_name); - curline -= len; - p += len; - + sbuf_printf(&sbuf, ","); + sbuf_printf(&sbuf, "%s", + kmemzones[i].kz_name); first = 0; } } - - len = snprintf(p, 2, "\n"); - p += len; + sbuf_printf(&sbuf, "\n"); } + sbuf_finish(&sbuf); + mtx_unlock(&malloc_mtx); - mtx_unlock(&malloc_mtx); - error = SYSCTL_OUT(req, buf, p - buf); + error = SYSCTL_OUT(req, sbuf_data(&sbuf), sbuf_len(&sbuf)); + sbuf_delete(&sbuf); free(buf, M_TEMP); return (error); } @@ -696,6 +730,7 @@ sysctl_kern_mprof(SYSCTL_HANDLER_ARGS) { int linesize = 64; + struct sbuf sbuf; uint64_t count; uint64_t waste; uint64_t mem; @@ -704,7 +739,6 @@ char *buf; int rsize; int size; - char *p; int len; int i; @@ -714,34 +748,30 @@ waste = 0; mem = 0; - p = buf = (char *)malloc(bufsize, M_TEMP, M_WAITOK|M_ZERO); - len = snprintf(p, bufsize, + buf = (char *)malloc(bufsize, M_TEMP, M_WAITOK|M_ZERO); + sbuf_new(&sbuf, buf, bufsize, SBUF_FIXEDLEN); + sbuf_printf(&sbuf, "\n Size Requests Real Size\n"); - bufsize -= len; - p += len; - for (i = 0; i < KMEM_ZSIZE; i++) { size = i << KMEM_ZSHIFT; rsize = kmemzones[kmemsize[i]].kz_size; count = (long long unsigned)krequests[i]; - len = snprintf(p, bufsize, "%6d%28llu%11d\n", - size, (unsigned long long)count, rsize); - bufsize -= len; - p += len; + sbuf_printf(&sbuf, "%6d%28llu%11d\n", size, + (unsigned long long)count, rsize); if ((rsize * count) > (size * count)) waste += (rsize * count) - (size * count); mem += (rsize * count); } - - len = snprintf(p, bufsize, + sbuf_printf(&sbuf, "\nTotal memory used:\t%30llu\nTotal Memory wasted:\t%30llu\n", (unsigned long long)mem, (unsigned long long)waste); - p += len; + sbuf_finish(&sbuf); - error = SYSCTL_OUT(req, buf, p - buf); + error = SYSCTL_OUT(req, sbuf_data(&sbuf), sbuf_len(&sbuf)); + sbuf_delete(&sbuf); free(buf, M_TEMP); return (error); } --- //depot/vendor/freebsd/src/sys/sys/malloc.h 2005/01/07 02:32:16 +++ //depot/user/rwatson/percpu/sys/sys/malloc.h 2005/04/30 14:37:07 @@ -50,25 +50,79 @@ #define M_MAGIC 877983977 /* time when first defined :-) */ +#ifdef WANT_MALLOC_TYPE_INTERNAL +/* + * Two malloc type structures are present: malloc_type, which is used by a + * type owner to declare the type, and malloc_type_internal, which holds + * malloc-owned statistics and other ABI-sensitive fields, such as the set of + * malloc statistics indexed by the compile-time MAXCPU constant. + * + * The malloc_type ks_next field is protected by malloc_mtx. Other fields in + * malloc_type are static after initialization so unsynchronized. + * + * Statistics in malloc_type_stats are written only when holding a critical + * section, but read lock-free resulting in possible (minor) races, which the + * monitoring app should take into account. + */ +struct malloc_type_stats { + u_long mts_memalloced; /* Bytes allocated on CPU. */ + u_long mts_memfreed; /* Bytes freed on CPU. */ + u_long mts_numallocs; /* Number of allocates on CPU. */ + u_long mts_numfrees; /* number of frees on CPU. */ + u_long mts_size; /* Bitmask of sizes allocated on CPU. */ +}; + +struct malloc_type_internal { + struct malloc_type_stats mti_stats[MAXCPU]; +}; +#endif + +/* + * ABI-compatible version of the old 'struct malloc_type', only all stats are + * now malloc-managed in malloc-owned memory rather than in caller memory, so + * as to avoid ABI issues. The ks_next pointer is reused as a pointer to the + * internal data handle. + * + * XXXRW: Why is this not ifdef _KERNEL? + * + * XXXRW: Use of ks_shortdesc has leaked out of kern_malloc.c. + */ struct malloc_type { - struct malloc_type *ks_next; /* next in list */ - u_long ks_memuse; /* total memory held in bytes */ - u_long ks_size; /* sizes of this thing that are allocated */ - u_long ks_inuse; /* # of packets of this type currently in use */ - uint64_t ks_calls; /* total packets of this type ever allocated */ - u_long ks_maxused; /* maximum number ever used */ - u_long ks_magic; /* if it's not magic, don't touch it */ - const char *ks_shortdesc; /* short description */ - struct mtx ks_mtx; /* lock for stats */ + struct malloc_type *ks_next; /* Next in global chain. */ + u_long _ks_memuse; /* No longer used. */ + u_long _ks_size; /* No longer used. */ + u_long _ks_inuse; /* No longer used. */ + uint64_t _ks_calls; /* No longer used. */ + u_long _ks_maxused; /* No longer used. */ + u_long ks_magic; /* Detect programmer error. */ + const char *ks_shortdesc; /* Printable type name. */ + + /* + * struct malloc_type was terminated with a struct mtx, which is no + * longer required. For ABI reasons, continue to flesh out the full + * size of the old structure, but reuse the _lo_class field for our + * internal data handle. + */ + void *ks_handle; /* Priv. data, was lo_class. */ + const char *_lo_name; + const char *_lo_type; + u_int _lo_flags; + void *_lo_list_next; + struct witness *_lo_witness; + uintptr_t _mtx_lock; + u_int _mtx_recurse; }; #ifdef _KERNEL -#define MALLOC_DEFINE(type, shortdesc, longdesc) \ - struct malloc_type type[1] = { \ - { NULL, 0, 0, 0, 0, 0, M_MAGIC, shortdesc, {} } \ - }; \ - SYSINIT(type##_init, SI_SUB_KMEM, SI_ORDER_SECOND, malloc_init, type); \ - SYSUNINIT(type##_uninit, SI_SUB_KMEM, SI_ORDER_ANY, malloc_uninit, type) +#define MALLOC_DEFINE(type, shortdesc, longdesc) \ + struct malloc_type type[1] = { \ + { NULL, 0, 0, 0, 0, 0, M_MAGIC, shortdesc, NULL, NULL, \ + NULL, 0, NULL, NULL, 0, 0 } \ + }; \ + SYSINIT(type##_init, SI_SUB_KMEM, SI_ORDER_SECOND, malloc_init, \ + type); \ + SYSUNINIT(type##_uninit, SI_SUB_KMEM, SI_ORDER_ANY, \ + malloc_uninit, type); #define MALLOC_DECLARE(type) \ extern struct malloc_type type[1] From owner-freebsd-performance@FreeBSD.ORG Mon May 2 12:45:55 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF44B16A4CF for ; Mon, 2 May 2005 12:45:55 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 07BC043D5D for ; Mon, 2 May 2005 12:45:55 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j42CjslL004639; Mon, 2 May 2005 07:45:54 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <427620C6.5020109@centtech.com> Date: Mon, 02 May 2005 07:44:54 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mariusz Grad References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <20050502121633.GA14896@wask.wask.wroc.pl> In-Reply-To: <20050502121633.GA14896@wask.wask.wroc.pl> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/861/Sat Apr 30 04:28:52 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 12:45:55 -0000 Mariusz Grad wrote: > Mike Tancsa: > >>A somewhat obvious question to some perhaps, but what server application >>mix on FreeBSD today sees an improvement using 64bit CPUs ? In my ISP >>centric world, my big apps are BIND, IMAP/POP3, httpd via apache, SMTP, AV >>and SPAM scanning, and firewalls/routing. Apart from larger RAM, why would >>these benefit from the 64bit world ? Or would they ? > > Benefits from AMD64: > - larger RAM limit (40bit memory address), > - 64bit GRPs, > - much faster access to memory (memory controller inside core - no northbridge), > - it much better scales (with many CPUs) (he doesnt share memory bandwith), > - large caches L1 = 64KB, L2 = 1MB (comparing to xeon 12/512K), Some Xeon's have 2MB now.. > - DEP = Data Execution Protection (it has to be supported via OS), Intel has the 'NX' (no execute) bit - isn't this the same? > - Multimedia Extensions for graphics / vector processing. Xeon's have this too I believe (I don't think this is anything new either). > = Opterons remove memory bottlenecks espacially with multi-cpus. > > Ive had large acceleration at: > - math computation (f77, 2-4GB double precision matrix) single opteron 1.8GHz (sun v20z) was 4x faster then celeron 2.4GHz. > - databases (pgsql) Opteron 1.8 GHz 2x faster then Xeon 3.2 That's interesting, although I wouldn't compare a Celeron against an AMD64, but interesting nonetheless. > I would assume that everything which can run: > - in parallel, > - on multi-Opterons CPU, > - which uses heavly RAM, > - which can take benefits from 64bit registers (floating points), > will have _extreamly_ boost comparing to Intels EMT64 (or Itanium2). Agreed.. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon May 2 12:48:36 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2E42516A4CF; Mon, 2 May 2005 12:48:36 +0000 (GMT) Received: from ms1.as.pvp.se (dns.pvp.se [213.64.187.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7C54443D41; Mon, 2 May 2005 12:48:35 +0000 (GMT) (envelope-from kama@pvp.se) Received: by ms1.as.pvp.se (Postfix, from userid 1001) id 5701EA7; Mon, 2 May 2005 14:48:33 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by ms1.as.pvp.se (Postfix) with ESMTP id 52C16A6; Mon, 2 May 2005 14:48:33 +0200 (CEST) Date: Mon, 2 May 2005 14:48:33 +0200 (CEST) From: kama X-X-Sender: kama@ns1.as.pvp.se To: Eric Anderson In-Reply-To: <42761FA7.1030500@centtech.com> Message-ID: <20050502144658.H22614@ns1.as.pvp.se> References: <20050501112351.4184.qmail@web41205.mail.yahoo.com> <42761FA7.1030500@centtech.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-performance@freebsd.org cc: Robert Watson cc: =?ISO-8859-15?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 12:48:36 -0000 On Mon, 2 May 2005, Eric Anderson wrote: > I'll be honest here, I don't care much if the speed difference between > 4.X and 5.X is measureable, or whatever. What I find is a little > telling of an issue somewhere, is that READS are slower than WRITES! > This is totally bogus to me - dd'ing a file to a filesystem, then > umounting should take longer than dd'ing from the disk to /dev/null, it > nearly every config I can dream up. Maybe it's the speed at which > /dev/null can gobble bits (seems highly unlikely!), or maybe GEOM is > busy doing a check or some routine to data being accessed directly from > the disk device instead of through a filesystem? I don't know, but it > is an issue, and I'm sure we'll get nailed up to a fence in some > benchmark somewhere if we don't fix it.. dev-null is not the issue... my own written testprogram that only read up data to a buffer in memory showed the same results as doing a dd to dev-null. /Bjorn From owner-freebsd-performance@FreeBSD.ORG Mon May 2 12:51:29 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 51ED616A4CE; Mon, 2 May 2005 12:51:29 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1F4FF43D3F; Mon, 2 May 2005 12:51:28 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42CpNlp017337; Mon, 2 May 2005 14:51:23 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Eric Anderson From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 07:40:07 CDT." <42761FA7.1030500@centtech.com> Date: Mon, 02 May 2005 14:51:23 +0200 Message-ID: <17336.1115038283@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: freebsd-performance@freebsd.org cc: Robert Watson cc: =?ISO-8859-15?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 12:51:29 -0000 In message <42761FA7.1030500@centtech.com>, Eric Anderson writes: > >I'll be honest here, I don't care much if the speed difference between >4.X and 5.X is measureable, or whatever. What I find is a little >telling of an issue somewhere, is that READS are slower than WRITES! >This is totally bogus to me - [...] If the disk has bad sectors or other hardware issues, this is not an atypical result. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:10:23 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A979916A4CE; Mon, 2 May 2005 13:10:23 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id B135843D53; Mon, 2 May 2005 13:10:22 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001375491.msg; Mon, 02 May 2005 14:06:00 +0100 Message-ID: <003801c54f18$2d4dccd0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Eric Anderson" , "Poul-Henning Kamp" References: <17336.1115038283@critter.freebsd.dk> Date: Mon, 2 May 2005 14:09:27 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 14:06:00 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 14:06:01 +0100 cc: freebsd-performance@freebsd.org cc: Robert Watson cc: =?iso-8859-1?Q?=22Arne_\=22W=F6rner\=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:10:23 -0000 Its highly unlikely that the 4 people on different hardware that have tested this all have disks with bad sectors. I've just finished doing a full battery of tests across: FreeBSD: 4.11-RELEASE, 5.4-STABLE, 6.0-CURRENT, Suse 9.1 I'll post the results soon but suffice to say the results for FreeBSD don't look good. Steve ----- Original Message ----- From: "Poul-Henning Kamp" > In message <42761FA7.1030500@centtech.com>, Eric Anderson writes: >> >>I'll be honest here, I don't care much if the speed difference between >>4.X and 5.X is measureable, or whatever. What I find is a little >>telling of an issue somewhere, is that READS are slower than WRITES! >>This is totally bogus to me - [...] > > If the disk has bad sectors or other hardware issues, this is not > an atypical result. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:11:53 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CC0F216A4CE; Mon, 2 May 2005 13:11:53 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C58043D53; Mon, 2 May 2005 13:11:53 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j42DBqTI004825; Mon, 2 May 2005 08:11:52 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <427626DC.5030702@centtech.com> Date: Mon, 02 May 2005 08:10:52 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Poul-Henning Kamp References: <17336.1115038283@critter.freebsd.dk> In-Reply-To: <17336.1115038283@critter.freebsd.dk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/862/Mon May 2 07:24:27 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org cc: Robert Watson cc: =?ISO-8859-15?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:11:53 -0000 Poul-Henning Kamp wrote: > In message <42761FA7.1030500@centtech.com>, Eric Anderson writes: > >>I'll be honest here, I don't care much if the speed difference between >>4.X and 5.X is measureable, or whatever. What I find is a little >>telling of an issue somewhere, is that READS are slower than WRITES! >>This is totally bogus to me - [...] > > > If the disk has bad sectors or other hardware issues, this is not > an atypical result. Don't mean to be terse here, but I'm talking about the same test done an two different RAID5 configurations, with different disks, and not just me - other users in this very thread see the same issue.. I'm not an idiot, with a stupid problem here - this is a real problem, not a user created one. It isn't catastrophic, but come on - 2x performance difference between reads and writes? I wouldn't think twice if writes were 50% speed as reads, but it's the other way around. I'm tired of beating this with a stick - why doesn't anyone just trust the fact that something isn't right, and try to help find the problem and fix it? I'm not familiar with debugging the kernel, or source code enough to find it by myself. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:15:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9AC0E16A4CF; Mon, 2 May 2005 13:15:10 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 91EB643D58; Mon, 2 May 2005 13:15:09 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42DF63E017443; Mon, 2 May 2005 15:15:06 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Eric Anderson From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 08:10:52 CDT." <427626DC.5030702@centtech.com> Date: Mon, 02 May 2005 15:15:06 +0200 Message-ID: <17442.1115039706@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: freebsd-performance@freebsd.org cc: Robert Watson cc: =?ISO-8859-15?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:15:10 -0000 In message <427626DC.5030702@centtech.com>, Eric Anderson writes: >Don't mean to be terse here, but I'm talking about the same test done an >two different RAID5 configurations, with different disks, and not just >me - other users in this very thread see the same issue.. Uhm, if you are using RAID5 and your requests are not aligned and sized after the RAID5 you should *expect* read performance to be poor. If you your request ends up accessing two different blocks even just once per stripe, this totally kills performance. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:17:13 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BD34416A4D1 for ; Mon, 2 May 2005 13:17:13 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1ECE543D45 for ; Mon, 2 May 2005 13:17:13 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j42DHC6w004850; Mon, 2 May 2005 08:17:12 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <4276281C.6060209@centtech.com> Date: Mon, 02 May 2005 08:16:12 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Poul-Henning Kamp References: <17442.1115039706@critter.freebsd.dk> In-Reply-To: <17442.1115039706@critter.freebsd.dk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/862/Mon May 2 07:24:27 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:17:13 -0000 Poul-Henning Kamp wrote: > In message <427626DC.5030702@centtech.com>, Eric Anderson writes: > > >>Don't mean to be terse here, but I'm talking about the same test done an >>two different RAID5 configurations, with different disks, and not just >>me - other users in this very thread see the same issue.. > > > Uhm, if you are using RAID5 and your requests are not aligned and > sized after the RAID5 you should *expect* read performance to be poor. > > If you your request ends up accessing two different blocks even just > once per stripe, this totally kills performance. Wouldn't this be a problem for writes then too? -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:21:04 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0426816A4CE; Mon, 2 May 2005 13:21:04 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 13DF443D31; Mon, 2 May 2005 13:21:03 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [195.163.185.142] (i2-142.rommon.fi [195.163.185.142]) by silver.he.iki.fi (Postfix) with ESMTP id 36303BC46; Mon, 2 May 2005 16:21:00 +0300 (EEST) Message-ID: <42762960.4020600@he.iki.fi> Date: Mon, 02 May 2005 16:21:36 +0300 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: kama References: <20050501112351.4184.qmail@web41205.mail.yahoo.com> <42761FA7.1030500@centtech.com> <20050502144658.H22614@ns1.as.pvp.se> In-Reply-To: <20050502144658.H22614@ns1.as.pvp.se> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: =?ISO-8859-1?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:21:04 -0000 kama wrote: > > > > >dev-null is not the issue... my own written testprogram that only read up >data to a buffer in memory showed the same results as doing a dd to >dev-null. > > > And dd from zero to null does: 114541264896 bytes transferred in 27.716454 secs (4132608911 bytes/sec) Pete >/Bjorn >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:22:52 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9CA0E16A4CE; Mon, 2 May 2005 13:22:52 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 50F3743D4C; Mon, 2 May 2005 13:22:52 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [195.163.185.142] (i2-142.rommon.fi [195.163.185.142]) by silver.he.iki.fi (Postfix) with ESMTP id 44A2ABC46; Mon, 2 May 2005 16:22:51 +0300 (EEST) Message-ID: <427629CB.6060706@he.iki.fi> Date: Mon, 02 May 2005 16:23:23 +0300 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Poul-Henning Kamp References: <17442.1115039706@critter.freebsd.dk> In-Reply-To: <17442.1115039706@critter.freebsd.dk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: =?ISO-8859-1?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:22:52 -0000 Poul-Henning Kamp wrote: >In message <427626DC.5030702@centtech.com>, Eric Anderson writes: > > > >>Don't mean to be terse here, but I'm talking about the same test done an >>two different RAID5 configurations, with different disks, and not just >>me - other users in this very thread see the same issue.. >> >> > >Uhm, if you are using RAID5 and your requests are not aligned and >sized after the RAID5 you should *expect* read performance to be poor. > >If you your request ends up accessing two different blocks even just >once per stripe, this totally kills performance. > > > > My tests were using RAID10 and just striping. (RAID0 might be the right name for it) Pete From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:23:01 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B2D6016A4CE for ; Mon, 2 May 2005 13:23:01 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id E98A843D4C for ; Mon, 2 May 2005 13:23:00 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42DMwKh017480; Mon, 2 May 2005 15:22:58 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Eric Anderson From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 08:16:12 CDT." <4276281C.6060209@centtech.com> Date: Mon, 02 May 2005 15:22:58 +0200 Message-ID: <17479.1115040178@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:23:01 -0000 In message <4276281C.6060209@centtech.com>, Eric Anderson writes: >Poul-Henning Kamp wrote: >> In message <427626DC.5030702@centtech.com>, Eric Anderson writes: >> >> >>>Don't mean to be terse here, but I'm talking about the same test done an >>>two different RAID5 configurations, with different disks, and not just >>>me - other users in this very thread see the same issue.. >> >> >> Uhm, if you are using RAID5 and your requests are not aligned and >> sized after the RAID5 you should *expect* read performance to be poor. >> >> If you your request ends up accessing two different blocks even just >> once per stripe, this totally kills performance. > >Wouldn't this be a problem for writes then too? I presume you would only compare read to write performance on a RAID5 device which has battery backed cache. Without a battery backed cache (or pretending to have one) RAID5 write performance is abysmall no matter which alignment you use. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:23:31 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F0C2E16A4CF; Mon, 2 May 2005 13:23:31 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 99E4843D39; Mon, 2 May 2005 13:23:31 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [195.163.185.142] (i2-142.rommon.fi [195.163.185.142]) by silver.he.iki.fi (Postfix) with ESMTP id B5854BC47; Mon, 2 May 2005 16:23:30 +0300 (EEST) Message-ID: <427629F7.5060704@he.iki.fi> Date: Mon, 02 May 2005 16:24:07 +0300 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Poul-Henning Kamp References: <17336.1115038283@critter.freebsd.dk> In-Reply-To: <17336.1115038283@critter.freebsd.dk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: =?ISO-8859-1?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:23:32 -0000 Poul-Henning Kamp wrote: >In message <42761FA7.1030500@centtech.com>, Eric Anderson writes: > > >>I'll be honest here, I don't care much if the speed difference between >>4.X and 5.X is measureable, or whatever. What I find is a little >>telling of an issue somewhere, is that READS are slower than WRITES! >>This is totally bogus to me - [...] >> >> > >If the disk has bad sectors or other hardware issues, this is not >an atypical result. > > > The drives I have report this using SMART and the counters stay at zero. Yes, we've seen the counters go nonzero when problems do exist. Pete From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:25:32 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BAF7916A4CE; Mon, 2 May 2005 13:25:32 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id F008143D49; Mon, 2 May 2005 13:25:31 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001375517.msg; Mon, 02 May 2005 14:21:36 +0100 Message-ID: <002201c54f1a$5c05dfc0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Eric Anderson" , "Poul-Henning Kamp" References: <17442.1115039706@critter.freebsd.dk> Date: Mon, 2 May 2005 14:25:04 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 14:21:36 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 14:21:37 +0100 cc: freebsd-performance@freebsd.org cc: Robert Watson cc: =?iso-8859-1?Q?=22Arne_\=22W=F6rner\=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:25:32 -0000 ----- Original Message ----- From: "Poul-Henning Kamp" >>Don't mean to be terse here, but I'm talking about the same test done an >>two different RAID5 configurations, with different disks, and not just >>me - other users in this very thread see the same issue.. > > Uhm, if you are using RAID5 and your requests are not aligned and > sized after the RAID5 you should *expect* read performance to be poor. > > If you your request ends up accessing two different blocks even just > once per stripe, this totally kills performance. Interesting stuff so: 1. How to we test if this is happening? 2. How do we prevent it from happening? 3. Why would this be effecting reads and not writes as surely the same blocking is being done for both? RAID5 is becoming more and more the norm with disk and controller prices dropping and as such I think it would be good if could get this. one sorted. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:25:44 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F11D716A4CE for ; Mon, 2 May 2005 13:25:44 +0000 (GMT) Received: from web41214.mail.yahoo.com (web41214.mail.yahoo.com [66.218.93.47]) by mx1.FreeBSD.org (Postfix) with SMTP id BD64D43D45 for ; Mon, 2 May 2005 13:25:44 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 73807 invoked by uid 60001); 2 May 2005 13:25:44 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=dLAUYbVn6cKuBRgQ0/0G4nfGd8qWARacZHkf54FUtff+KU4N7y0q2f+HJ10xoSoYjTQRWXMhEj1WiLdBYjLnAnWLyWJCf6LxS/luz1OfvjRvlgO3U8BCQzl9RM/jZ806Ue3Zqt8nmqIhAo1JJB6T97o4uI6KwqG85alCqLbbQSA= ; Message-ID: <20050502132544.73805.qmail@web41214.mail.yahoo.com> Received: from [83.129.171.204] by web41214.mail.yahoo.com via HTTP; Mon, 02 May 2005 06:25:44 PDT Date: Mon, 2 May 2005 06:25:44 -0700 (PDT) From: Arne "Wörner" To: Poul-Henning Kamp In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:25:45 -0000 --- Poul-Henning Kamp wrote: > Uhm, if you are using RAID5 and your requests are not aligned > and sized after the RAID5 you should *expect* read performance > to be poor. > Wouldn't that affect both (read and write) in the same way? > If the disk has bad sectors or other hardware issues, this is > not an atypical result. > dito :-) -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:28:54 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAFB416A4CE for ; Mon, 2 May 2005 13:28:54 +0000 (GMT) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0BEFB43D45 for ; Mon, 2 May 2005 13:28:54 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j42DSrFU074818; Mon, 2 May 2005 08:28:53 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42762AD8.1020607@centtech.com> Date: Mon, 02 May 2005 08:27:52 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Poul-Henning Kamp References: <17479.1115040178@critter.freebsd.dk> In-Reply-To: <17479.1115040178@critter.freebsd.dk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:28:54 -0000 Poul-Henning Kamp wrote: > In message <4276281C.6060209@centtech.com>, Eric Anderson writes: > >>Poul-Henning Kamp wrote: >> >>>In message <427626DC.5030702@centtech.com>, Eric Anderson writes: >>> >>> >>> >>>>Don't mean to be terse here, but I'm talking about the same test done an >>>>two different RAID5 configurations, with different disks, and not just >>>>me - other users in this very thread see the same issue.. >>> >>> >>>Uhm, if you are using RAID5 and your requests are not aligned and >>>sized after the RAID5 you should *expect* read performance to be poor. >>> >>>If you your request ends up accessing two different blocks even just >>>once per stripe, this totally kills performance. >> >>Wouldn't this be a problem for writes then too? > > > I presume you would only compare read to write performance on a RAID5 > device which has battery backed cache. > > Without a battery backed cache (or pretending to have one) RAID5 > write performance is abysmall no matter which alignment you use. If I write a 10GB file to disk (RAID array has 1GB cache, system has 1GB memory), then I should definitely see better read performance reading that same file back to /dev/null than writing it, right? How about this - you tell me what test to run, and I'll do it (as long as it doesn't destory my data). Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:29:52 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9821116A4CE for ; Mon, 2 May 2005 13:29:52 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 016C143D1F for ; Mon, 2 May 2005 13:29:52 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001375526.msg for ; Mon, 02 May 2005 14:25:16 +0100 Message-ID: <002701c54f1a$dde7b0e0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Eric Anderson" , "Poul-Henning Kamp" References: <17479.1115040178@critter.freebsd.dk> Date: Mon, 2 May 2005 14:28:42 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 14:25:16 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 14:25:17 +0100 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:29:52 -0000 ----- Original Message ----- From: "Poul-Henning Kamp" >>Wouldn't this be a problem for writes then too? > > I presume you would only compare read to write performance on a RAID5 > device which has battery backed cache. > > Without a battery backed cache (or pretending to have one) RAID5 > write performance is abysmall no matter which alignment you use. This not what's been reported we are seeing writes that are 2x the speed of reads. Give the additional overhead that writes encure on RAID5 this should never be the case. The results I go in my tests where: RAID5 ( 5 disk ) Read: ~50MB/s Write: ~140MB/s Single Disk ( same controller ): Read: ~50MB/s Write: ~50MB/s This just doesn't sit right with me something is seriously wrong. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:50:03 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 80AEC16A4CE; Mon, 2 May 2005 13:50:03 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 81A1B43D41; Mon, 2 May 2005 13:50:02 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42Dnw1g017797; Mon, 2 May 2005 15:49:58 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Petri Helenius From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 16:23:23 +0300." <427629CB.6060706@he.iki.fi> Date: Mon, 02 May 2005 15:49:58 +0200 Message-ID: <17796.1115041798@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: =?ISO-8859-1?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:50:03 -0000 In message <427629CB.6060706@he.iki.fi>, Petri Helenius writes: >> >My tests were using RAID10 and just striping. (RAID0 might be the right >name for it) Same thing applies, and it depends on how the reqeust alignment/size and stripe alignment/size interacts. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:52:52 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5882916A4D0 for ; Mon, 2 May 2005 13:52:52 +0000 (GMT) Received: from mail.rfnj.org (ns1.rfnj.org [66.180.172.156]) by mx1.FreeBSD.org (Postfix) with ESMTP id C43A343D58 for ; Mon, 2 May 2005 13:52:51 +0000 (GMT) (envelope-from asym@rfnj.org) Received: from megalomaniac.rfnj.org (ool-45736df1.dyn.optonline.net [69.115.109.241]) by mail.rfnj.org (Postfix) with ESMTP id 0AEDD2A3; Mon, 2 May 2005 09:52:51 -0400 (EDT) Message-Id: <6.2.1.2.2.20050502094757.037077f0@mail.rfnj.org> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Mon, 02 May 2005 09:58:49 -0400 To: "Steven Hartland" , "Eric Anderson" , "Poul-Henning Kamp" From: Allen In-Reply-To: <002701c54f1a$dde7b0e0$b3db87d4@multiplay.co.uk> References: <17479.1115040178@critter.freebsd.dk> <002701c54f1a$dde7b0e0$b3db87d4@multiplay.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:52:52 -0000 At 09:28 5/2/2005, Steven Hartland wrote: >----- Original Message ----- From: "Poul-Henning Kamp" >>>Wouldn't this be a problem for writes then too? >>I presume you would only compare read to write performance on a RAID5 >>device which has battery backed cache. >>Without a battery backed cache (or pretending to have one) RAID5 >>write performance is abysmall no matter which alignment you use. > >This not what's been reported we are seeing writes that are 2x the >speed of reads. Give the additional overhead that writes encure >on RAID5 this should never be the case. The results I go in my tests >where: FWIW I have exactly this situation on a Mylex controller I have, if I enable the write cache on the card. For some reason or another, doing so results in marginally improved write speeds, but sustained read speeds that drop well below 1/4 of the values they get when the *write* cache is disabled. I have never figured out nor discovered a satisfactory explaination for why this is the case, and unlike Erics situation, this is consistent for me on that card across all OSes and tests. May be some more fuel for the fire however, to someone in the know. Also you should keep in mind, there could simply be some really goofy controller option enabled, that forces the RAID5 to behave in a "degraded" state for reads -- forcing it to read up all the other disks in the stripe and calculate the XOR again, to make sure the data it read off the disk matches the checksum. It's rare, but I've seen it before, and it will cause exactly this sort of RAID5 performance inversion. Since the XOR is recalculated on every write and requires only reading up one sector on a different disk, options that do the above will result in read scores drastically lower than writes to the same array. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:53:39 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2DBD616A4CE; Mon, 2 May 2005 13:53:39 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 65DE243D58; Mon, 2 May 2005 13:53:38 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42DrYHn017814; Mon, 2 May 2005 15:53:34 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: "Steven Hartland" From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 14:25:04 BST." <002201c54f1a$5c05dfc0$b3db87d4@multiplay.co.uk> Date: Mon, 02 May 2005 15:53:34 +0200 Message-ID: <17813.1115042014@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:53:39 -0000 >Interesting stuff so: >1. How to we test if this is happening? Calculate by hand what the offset of the striped/raid part of the disk is (ie: take slice+partition stats into account). >2. How do we prevent it from happening? Make sure that the first sector of a partition/slice is always the first sector in a stripe on your raid/stripe/whatever. >3. Why would this be effecting reads and not writes as surely the same >blocking is being done for both? Write on RAID5 uses a cache which lies to you about when things are safely stored on the disk. Good RAID5 has battery backup for that cache. The MBR slice format is stupid because it more often than not gets this exactly wrong. Typically there are 63 "sectors per track" and that ruins any alignment in 99% of the cases. Sysinstall, fdisk and bsdlabel should know about all this and try to help the user get it right. Fixing them to do so may be more trouble than writing a better too bottom up. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 13:55:02 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 78B6516A4CF for ; Mon, 2 May 2005 13:55:02 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id AFE0043D46 for ; Mon, 2 May 2005 13:55:01 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42DsxwU017828; Mon, 2 May 2005 15:54:59 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Eric Anderson From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 08:27:52 CDT." <42762AD8.1020607@centtech.com> Date: Mon, 02 May 2005 15:54:59 +0200 Message-ID: <17827.1115042099@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 13:55:02 -0000 In message <42762AD8.1020607@centtech.com>, Eric Anderson writes: >If I write a 10GB file to disk (RAID array has 1GB cache, system has 1GB >memory), then I should definitely see better read performance reading >that same file back to /dev/null than writing it, right? Nope, quite the contrary: you will get much better write performance because your write operation finishes the moment it has hit the cache, but your read request only comes fast if it happens to be in the cache. With a 1GB cache and a 10GB file, this clearly will not be the case unless some kind of very good read-ahead happens in the RAID5 unit. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:10:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DB30216A4CE for ; Mon, 2 May 2005 14:10:10 +0000 (GMT) Received: from web41212.mail.yahoo.com (web41212.mail.yahoo.com [66.218.93.45]) by mx1.FreeBSD.org (Postfix) with SMTP id 866FD43D53 for ; Mon, 2 May 2005 14:10:10 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 89694 invoked by uid 60001); 2 May 2005 14:10:10 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=DpBZ/Zb468zvCa8Ut8x+egFLgUNnFXOVDA7r5CtafkoG+8dd6U8eBb9NEACICcKWgo4vAWwsLL/jNyiPRviOl1HjvkJ274QHj/K5ISxOS62BKjCsLsB51zkBXAXyh9RPHvjbt2daD8VOe3AQWYiz0zptu+v4Ae8zqn0uzNkNzfw= ; Message-ID: <20050502141010.89692.qmail@web41212.mail.yahoo.com> Received: from [83.129.199.224] by web41212.mail.yahoo.com via HTTP; Mon, 02 May 2005 07:10:10 PDT Date: Mon, 2 May 2005 07:10:10 -0700 (PDT) From: Arne "Wörner" To: Eric Anderson In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:10:11 -0000 --- Eric Anderson wrote: > Arne Wörner wrote: > > --- Robert Watson wrote: > >>On Sat, 30 Apr 2005, Arne Wörner wrote: > >> > >>>3. The man page geom(4) of R5.3 says "The GEOM framework > >>> provides an infrastructure in which "classes" can per- > >>> form transformations on disk I/O requests on their path > >>> from the upper kernel to the device drivers and back. > >>> > >>>Could it be, that geom slows something down (in some boxes > >>>the reading > >>>ops are very slow; in my box the writing ops are very slow)? > >> > >>[...] > >>against a further offset region of ad0. If they're against > >>the same bits of disk, the main difference here will be the > >>additional processing of the layers in the stack. A little > >>bit of math is required to figure out the offset, but dd > >>should be usable to figure out the incremental cost. > > > > I used the following commands: > > 1. dd if=/dev/ad0s2a bs=16k count=10000 of=/dev/null > > 2. dd if=/dev/ad0 iseek=2100357 bs=16k count=10000 > > of=/dev/null > > (I think I did the math correctly; indeed the read speed of my > > ad0 varies between 45MB/sec (iseek=0) and 25MB/sec (in the > > end) with bs=128k). > > The results were nearly the same (both between 26MB/sec and > > 28MB/sec). Maybe I should have done it in single user mode. > > > > My other hard disc ad1 (it is newer and bigger and faster and > > more furious) behaves better (but I can just try writing via > > the file system (ufs+s) and a final sync): The sync just > > needs .24sec of 18.28 seconds; the read and write speed > > is nearly the same (about 37MB/sec+/-1MB/sec). > > > > Is there anything else, that could help to find the reason for > > the difference between ad0's speed in R4.11 and R5.3? > > I'll be honest here, I don't care much if the speed difference > between 4.X and 5.X is measureable, or whatever. > I understand that. I believe, that it is very interesting, that thinks might have been better before (some years ago I learned of evolutionary algorithms; the development of an operating system could be seen as a big evolutionary algorithm, that is executed in human brains...). At least those comparisons could help to find evidence, that it could be better than it is, because it was better already before... :-) Does somebody feel me? > What I find is a little telling of an issue somewhere, is > that READS are slower than WRITES! > In my case it is vice versa (at least ad0). In ad1 it is like I would expect it. And Mr. Watson found another case, where read AND write throughput looked reasonable (I do not know the thread anymore; it was last year maybe...). > This is totally bogusto me - dd'ing a file to a filesystem, > then umounting should take longer than dd'ing from the disk > to /dev/null, it nearly every config I can dream up. > Yup! I agree. > Maybe it's the speed at which /dev/null can gobble bits > (seems highly unlikely!), or maybe GEOM is busy doing a > check or some routine to data being accessed directly from > the disk device instead of through a filesystem? > Maybe. But why are the symptoms are so different? The symptoms seem to depend on the hard disc/controller manufacturer... Maybe there are different reasons for the suboptimal behaviour. SEAGATE ST340015A/3.01 R4.11 reads and writes at same speed from/to /dev/ad0s2e R5.3 reads 5x faster than it writes from/to /dev/ad0s2e R4.11+R5.3 read at same speed from /dev/ad0 R5.3 writes 5 times slower than it reads from ufs+s filesystem SAMSUNG SP1604N/TM100-24 R4.11+R5.3 read at same speed from /dev/ad1 (appr. 60MB/sec) (although 80MB/sec is what the disc can do?) R5.3 writes and reads at or about at the same rate from ufs+s filesystem (read is 10% slower; write needs a long (256MB bs=128k) read from /dev/ad1 in order to reach the maximum fs-write-speed of 66% of the /dev/ad1-write-speed) So there are the following questions in my setting: 1. Why does my filesystems on /dev/ad1 need a long read before they perform with 66% of the "max" (initially they just perform with 20% of the "max")? 2. Why does the Seagate disc write 5 times slower than it reads (but just with R5.3)? Did you ever try dd if=/dev/zero of=/dev/null bs=1m count=10000 ? My slightly busy system is done with it after 38secs (appr. 270MBytes/sec. > I don't know, but it is an issue, and I'm sure we'll get > nailed up to a fence in some benchmark somewhere if we > don't fix it.. > Yup! -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:14:56 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C93D316A4CE for ; Mon, 2 May 2005 14:14:56 +0000 (GMT) Received: from web41212.mail.yahoo.com (web41212.mail.yahoo.com [66.218.93.45]) by mx1.FreeBSD.org (Postfix) with SMTP id 8E7CF43D2D for ; Mon, 2 May 2005 14:14:56 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 90375 invoked by uid 60001); 2 May 2005 14:14:56 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=GlUbtqqXBnKIeDmVFOBf0dTY3l3cSyLTRlf3ycZJp2sITHqIhfeSQH2MEDuQd5JiX6qSqE8LAWXgtGsEjGHVXexi0RNZpYBW57gNwyJTkhvkaM1VZ92pih1kv68QUorkEGNQUKxZdwYJQI23S61O5stGMe2o/uM3KaInhzvtoOk= ; Message-ID: <20050502141456.90371.qmail@web41212.mail.yahoo.com> Received: from [83.129.199.224] by web41212.mail.yahoo.com via HTTP; Mon, 02 May 2005 07:14:56 PDT Date: Mon, 2 May 2005 07:14:56 -0700 (PDT) From: Arne "Wörner" To: Allen In-Reply-To: <6.2.1.2.2.20050502094757.037077f0@mail.rfnj.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:14:56 -0000 --- Allen wrote: > Also you should keep in mind, there could simply be some really > goofy > controller option enabled, that forces the RAID5 to behave in a > "degraded" > state for reads -- forcing it to read up all the other disks in > the stripe > and calculate the XOR again, to make sure the data it read off > the disk > matches the checksum. It's rare, but I've seen it before, and > it will > cause exactly this sort of RAID5 performance inversion. Since > the XOR is > recalculated on every write and requires only reading up one > sector on a > different disk, options that do the above will result in read > scores > drastically lower than writes to the same array. > Isn't that compensated by the cache? I mean: We would just 1. read all the blocks, that correspond to the block, that is requested, 2. put them all into the cache 3. check the parity bits (XOR should be very fast; especially in comparison to the disc read times) 4. keep them in the cache (some kind of read ahead...) 5. send the requested block to the driver -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:22:08 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AEAF716A4CE for ; Mon, 2 May 2005 14:22:08 +0000 (GMT) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5EBF043D1D for ; Mon, 2 May 2005 14:22:08 +0000 (GMT) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id 7A19EFB43; Mon, 2 May 2005 10:22:07 -0400 (EDT) Received: by canoe.dclg.ca (Postfix, from userid 101) id CF03F1A09CD; Mon, 2 May 2005 10:22:04 -0400 (EDT) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17014.14219.963494.753741@canoe.dclg.ca> Date: Mon, 2 May 2005 10:22:03 -0400 To: Mike Tancsa In-Reply-To: <6.2.1.2.0.20050501193659.05483bd0@64.7.153.2> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <42750861.8000509@mac.com> <6.2.1.2.0.20050501193659.05483bd0@64.7.153.2> X-Mailer: VM 7.17 under 21.4 (patch 17) "Jumbo Shrimp" XEmacs Lucid cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:22:08 -0000 >>>>> "Mike" == Mike Tancsa writes: Mike> Thanks! I was worried I was missing something obvious in all Mike> the 64bit excitement. For my apps where I cant balance across Mike> multiple machines or where space is an issue, dual core CPUs I Mike> think might be more interesting to look at in a few months. Hi Mike! Anyways, we've been finding that raw packet passing is significantly better on some of the newer tyan opteron boards. But then that may have as much to do with them shipping with multiple independant PCI-X slots than the opterons themselves. Memory bandwidth probably doesn't hurt, either. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can only be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:26:19 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E9B4016A4CE for ; Mon, 2 May 2005 14:26:19 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id CCF6643D55 for ; Mon, 2 May 2005 14:26:18 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42EQCED018111; Mon, 2 May 2005 16:26:12 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Allen From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 09:58:49 EDT." <6.2.1.2.2.20050502094757.037077f0@mail.rfnj.org> Date: Mon, 02 May 2005 16:26:12 +0200 Message-ID: <18110.1115043972@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Steven Hartland Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:26:20 -0000 In message <6.2.1.2.2.20050502094757.037077f0@mail.rfnj.org>, Allen writes: I just want to add: This is why I really would love for us to have a real RAID3 implemetation. RAID3 is not commercially viable because windows cannot use non-512 byte sectors. We can. RAID3 would scream for us. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:29:24 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B4CE816A4CE for ; Mon, 2 May 2005 14:29:24 +0000 (GMT) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 829BC43D1D for ; Mon, 2 May 2005 14:29:23 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j42ETMcm005516; Mon, 2 May 2005 09:29:22 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42763906.1040202@centtech.com> Date: Mon, 02 May 2005 09:28:22 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050325 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Poul-Henning Kamp References: <18110.1115043972@critter.freebsd.dk> In-Reply-To: <18110.1115043972@critter.freebsd.dk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/862/Mon May 2 07:24:27 2005 on mh1.centtech.com X-Virus-Status: Clean cc: freebsd-performance@freebsd.org cc: Allen cc: Steven Hartland Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:29:24 -0000 Poul-Henning Kamp wrote: > In message <6.2.1.2.2.20050502094757.037077f0@mail.rfnj.org>, Allen writes: > > > I just want to add: This is why I really would love for us to have > a real RAID3 implemetation. > > RAID3 is not commercially viable because windows cannot use non-512 > byte sectors. > > We can. > > RAID3 would scream for us. What about disk arrays that support RAID3? -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology A lost ounce of gold may be found, a lost moment of time never. ------------------------------------------------------------------------ From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:33:21 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4A08716A4CE for ; Mon, 2 May 2005 14:33:21 +0000 (GMT) Received: from mail.rfnj.org (ns1.rfnj.org [66.180.172.156]) by mx1.FreeBSD.org (Postfix) with ESMTP id D3A6043D5F for ; Mon, 2 May 2005 14:33:20 +0000 (GMT) (envelope-from asym@rfnj.org) Received: from megalomaniac.rfnj.org (ool-45736df1.dyn.optonline.net [69.115.109.241]) by mail.rfnj.org (Postfix) with ESMTP id 91E5919D; Mon, 2 May 2005 10:33:21 -0400 (EDT) Message-Id: <6.2.1.2.2.20050502103041.037618d0@mail.rfnj.org> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Mon, 02 May 2005 10:39:18 -0400 To: Arne =?iso-8859-1?Q?=22W=F6rner=22?= From: Allen In-Reply-To: <20050502141456.90371.qmail@web41212.mail.yahoo.com> References: <6.2.1.2.2.20050502094757.037077f0@mail.rfnj.org> <20050502141456.90371.qmail@web41212.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:33:21 -0000 At 10:14 5/2/2005, Arne "W=F6rner" wrote: >--- Allen wrote: > > Also you should keep in mind, there could simply be some really > > goofy > > controller option enabled, that forces the RAID5 to behave in a > > "degraded" > > state for reads -- forcing it to read up all the other disks in > > the stripe > > and calculate the XOR again, to make sure the data it read off > > the disk > > matches the checksum. It's rare, but I've seen it before, and > > it will > > cause exactly this sort of RAID5 performance inversion. Since > > the XOR is > > recalculated on every write and requires only reading up one > > sector on a > > different disk, options that do the above will result in read > > scores > > drastically lower than writes to the same array. > > >Isn't that compensated by the cache? I mean: >We would just >1. read all the blocks, that correspond to the block, that is >requested, >2. put them all into the cache >3. check the parity bits (XOR should be very fast; especially in >comparison to the disc read times) >4. keep them in the cache (some kind of read ahead...) >5. send the requested block to the driver Your steps are appropriate but you should note that #3 is not true on cards= =20 that support RAID5 but do not have hardware-XOR. Some of the very cheap=20 i960 based cards have this failing, so the XOR itself is slow on top of=20 everything else. However, the cache doesn't play a part in this at all. It's the difference= =20 between these two read cycles, assume just one block was written to out of= =20 4, on a 5-disk system. Scenario A, verified read disabled: 1. RAID card reads up one block from appropriate drive. Done. Scenario B, verified read enabled: 1. RAID card reads up ALL blocks in the stripe (5 reads). 2. RAID card pretends the block requested is on a "degraded" drive, and=20 calculates it from the other 3 + the XOR stripe. 3. RAID card reports the value back, or tosses some kind of error. You can see, the cache just doesn't play a part in what I was describing,=20 which is basically the array performing as though it is degraded when in=20 fact it is not, to catch failures that would otherwise be missed. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:37:40 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E229816A4CF for ; Mon, 2 May 2005 14:37:40 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 04A0C43D31 for ; Mon, 2 May 2005 14:37:40 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42EbaCf018195; Mon, 2 May 2005 16:37:36 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Eric Anderson From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 09:28:22 CDT." <42763906.1040202@centtech.com> Date: Mon, 02 May 2005 16:37:36 +0200 Message-ID: <18194.1115044656@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: freebsd-performance@freebsd.org cc: Allen cc: Steven Hartland Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:37:41 -0000 In message <42763906.1040202@centtech.com>, Eric Anderson writes: >> In message <6.2.1.2.2.20050502094757.037077f0@mail.rfnj.org>, Allen writes: >> >> I just want to add: This is why I really would love for us to have >> a real RAID3 implemetation. >> >> RAID3 is not commercially viable because windows cannot use non-512 >> byte sectors. >> >> We can. >> >> RAID3 would scream for us. > >What about disk arrays that support RAID3? Would work for me, but most of them are dumbed down when they do RAID3: they have to hard format the disks to 128 byte sector sizes and similar madness in order to support 512 bytes sectors on the RAID3 volume. Some of them use 512 byte sector disks and internal sectorsizes of N*512 bytes and use their battery-backed cache to pretend to have 512 bytes logical sectorsize. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:38:19 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC30816A4CE for ; Mon, 2 May 2005 14:38:19 +0000 (GMT) Received: from web41207.mail.yahoo.com (web41207.mail.yahoo.com [66.218.93.40]) by mx1.FreeBSD.org (Postfix) with SMTP id A0C3543D2D for ; Mon, 2 May 2005 14:38:19 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 49820 invoked by uid 60001); 2 May 2005 14:38:19 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=OpTsAhcyV3xP8XbNSCEsil7qFaLjwO3q8AEdWLjvj38oWUdO7AA5k71WmDxKNcFiZAWwLKRvaPnvp20GP9b3XnVdmV5vA/nxy22RTzwr3Q9us56hxAqN5k3yhYwRFCx/OiXkDT46Wd5pUT4O+9vomhDxbZt8OB5ktF49iKIXZDI= ; Message-ID: <20050502143819.49818.qmail@web41207.mail.yahoo.com> Received: from [83.129.199.224] by web41207.mail.yahoo.com via HTTP; Mon, 02 May 2005 07:38:19 PDT Date: Mon, 2 May 2005 07:38:19 -0700 (PDT) From: Arne "Wörner" To: Allen In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:38:20 -0000 --- Allen wrote: > Scenario B, verified read enabled: > 1. RAID card reads up ALL blocks in the stripe (5 reads). > 2. RAID card pretends the block requested is on a "degraded" > drive, and > calculates it from the other 3 + the XOR stripe. > 3. RAID card reports the value back, or tosses some kind of > error. > > You can see, the cache just doesn't play a part in what I was > describing, > which is basically the array performing as though it is degraded > when in > fact it is not, to catch failures that would otherwise be > missed. > That would be a funny implementation. Step one could be done mostly from cache in case of sequential reads from a device (like /dev/ad0s1f or so). In this thread we always looked at sequential reads, as far as I recall... -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Mon May 2 14:42:46 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D6A9816A4CF for ; Mon, 2 May 2005 14:42:46 +0000 (GMT) Received: from mail.rfnj.org (ns1.rfnj.org [66.180.172.156]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9C5F543D4C for ; Mon, 2 May 2005 14:42:46 +0000 (GMT) (envelope-from asym@rfnj.org) Received: from megalomaniac.rfnj.org (ool-45736df1.dyn.optonline.net [69.115.109.241]) by mail.rfnj.org (Postfix) with ESMTP id 9F2D919D; Mon, 2 May 2005 10:42:47 -0400 (EDT) Message-Id: <6.2.1.2.2.20050502104554.03703c10@mail.rfnj.org> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Mon, 02 May 2005 10:48:44 -0400 To: Arne =?iso-8859-1?Q?=22W=F6rner=22?= From: asym In-Reply-To: <20050502143819.49818.qmail@web41207.mail.yahoo.com> References: <20050502143819.49818.qmail@web41207.mail.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 14:42:46 -0000 At 10:38 5/2/2005, Arne "W=F6rner" wrote: >--- Allen wrote: > > Scenario B, verified read enabled: > > 1. RAID card reads up ALL blocks in the stripe (5 reads). > > 2. RAID card pretends the block requested is on a "degraded" > > drive, and > > calculates it from the other 3 + the XOR stripe. > > 3. RAID card reports the value back, or tosses some kind of > > error. > > > > You can see, the cache just doesn't play a part in what I was > > describing, > > which is basically the array performing as though it is degraded > > when in > > fact it is not, to catch failures that would otherwise be > > missed. > > >That would be a funny implementation. Step one could be done >mostly from cache in case of sequential reads from a device (like >/dev/ad0s1f or so). In this thread we always looked at sequential >reads, as far as I recall... It's not "funny" it's just something some people do to try to catch more=20 errors. RAID5 will not catch transient write errors by default. If there was say=20 noise on the bus, and so the wrong value was written to the disk after the= =20 XOR was performed, but the right XOR data was written to the disk. RAID5 would normally never catch such an error, unless you run it in a=20 verified mode where it always behaves as though it is degraded, which is=20 what I was describing. Why you would, I'm not sure. It would catch the errors I suppose, but=20 there's nothing it can do about them -- it can't know if the XOR data or=20 the actual data is corrupt. Detection without correction is better than=20 nothing though, and if performance isn't your real goal, you can turn on=20 such features in some cards. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 16:43:55 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 40BB116A4E3; Mon, 2 May 2005 16:43:55 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id B99ED43D54; Mon, 2 May 2005 16:43:53 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001375947.msg; Mon, 02 May 2005 17:39:47 +0100 Message-ID: <003901c54f36$0c64ad40$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Poul-Henning Kamp" References: <17813.1115042014@critter.freebsd.dk> Date: Mon, 2 May 2005 17:43:16 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 17:39:47 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 17:39:47 +0100 cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Robert Watson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 16:43:55 -0000 ----- Original Message ----- From: "Poul-Henning Kamp" >>Interesting stuff so: >>1. How to we test if this is happening? > > Calculate by hand what the offset of the striped/raid part of the disk > is (ie: take slice+partition stats into account). How's that done? An explained example would be good. >>3. Why would this be effecting reads and not writes as surely the same >>blocking is being done for both? > > Write on RAID5 uses a cache which lies to you about when things are > safely stored on the disk. Your assuming I have cache on the card I dont. So the question still remains. > Good RAID5 has battery backup for that cache. > > The MBR slice format is stupid because it more often than not gets > this exactly wrong. Typically there are 63 "sectors per track" and > that ruins any alignment in 99% of the cases. Surely if this is know about it would be something that should have been fixed ages ago as it will be crippling everyone. > Sysinstall, fdisk and bsdlabel should know about all this and try > to help the user get it right. Fixing them to do so may be more > trouble than writing a better too bottom up. Ok from what your saying it sounds like RAID on FreeBSD is useless apart to create large disks. Now to the damaging facts the results from my two days worth of testing: *FreeBSD 6.0-CURRENT* H/W RAID5 ( 5 disk ) 16kb Stripe Write: 137Mb/s Read: 131Mb/s *FreeBSD 5.4-STABLE* H/W RAID5 16kb Stripe Write: 138Mb/s Read: 130Mb/s H/W RAID5 32kb Stripe Write: 137Mb/s Read: 115Mb/s H/W RAID5 64kb Stripe (Default ) Write: 139Mb/s Read: 88Mb/s H/W RAID5 1M Stripe Write: 141Mb/s Read: 51Mb/s S/W RAID5 Default Stripe ( vinum ) Write: 6Mb/s Read: 23Mb/s *FreeBSD 4.11-RELEASE* H/W RAID5 16kb Strip Write: 138Mb/s Read: 130Mb/s *Linux ( Suse 9.1 )* H/W RAID5 16kb Stripe Write: 105Mb/s Read: 137Mb/s H/W RAID5 32kb Stripe Write: 112Mb/s Read: 182Mb/s H/W RAID5 64kb Stripe ( Default ) Write: 120Mb/s Read: 122Mb/s H/W RAID5 1M Stripe Write: 117Mb/s Read: 102Mb/s S/W RAID5 Default Stripe ( Linux RAID ) Write: 269Mb/s Read: 259Mb/s *Summary / Conclusions* 1. Linux on this controller disk set is significantly quicker using H/W RAID logging a max read rate of 182Mb/s compared to 131Mb/s for FreeBSD. 2. The version of FreeBSD used ( for this controller ) didn't have any significant difference on performance, they where all poor. 3. Software RAID in linux totally blows away all the other configurations logging a max sustained read rate of 259Mb/s and write of 269Mb/s which shows the disks / controller are capable of producing the expected good performance. In comparison FreeBSD's vinum is not even worth using. N.B. vinum's extremely poor performance could have been down to poor default config but there are no performance tuning details to be found in the docs. *Test method / Hardware* Dual 244 Opteron 2Gb ECC RAM Highpoint 1820a controller in a PCI-X 133Mhz Slot 5 x Seagate 400GB SATA disks Write ( 6Gb ): dd if=/dev/zero of=/mnt/testfile bs=64k count=100000 Read ( 6Gb ): dd if=/mnt/testfile of=/dev/null bs=64k count=100000 and the following to check if block size was affecting / to test a typical app read. /usr/bin/time -h cat /mnt/testfile > /dev/null All tests where done on a empty formatted partition 100Gb in size, on a freshly initialised RAID5 array. OS was on an independent disk off the motherboard ( not connected to the raid controller ). All partitions / file system creation was done using the OS default tool i.e. FreeBSD: sysinstall, Suse: yast Note: FreeBSD 4.11 sysinstall's label was not functional on this array so it was created with a manual disklabel + newfs ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 16:45:15 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F0B2116A4CE; Mon, 2 May 2005 16:45:15 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 486AB43D1F; Mon, 2 May 2005 16:45:15 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [193.64.42.134] (h86.vuokselantie10.fi [193.64.42.134]) by silver.he.iki.fi (Postfix) with ESMTP id 4AFCCBC46; Mon, 2 May 2005 19:45:10 +0300 (EEST) Message-ID: <4276593A.5040908@he.iki.fi> Date: Mon, 02 May 2005 19:45:46 +0300 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Poul-Henning Kamp References: <17796.1115041798@critter.freebsd.dk> In-Reply-To: <17796.1115041798@critter.freebsd.dk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: =?ISO-8859-1?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 16:45:16 -0000 Poul-Henning Kamp wrote: >In message <427629CB.6060706@he.iki.fi>, Petri Helenius writes: > > > >>My tests were using RAID10 and just striping. (RAID0 might be the right >>name for it) >> >> > >Same thing applies, and it depends on how the reqeust alignment/size and >stripe alignment/size interacts. > > > I'm using either 64k or 1M buffer size with dd if=/dev/zero and of=/dev/da1 Pete From owner-freebsd-performance@FreeBSD.ORG Mon May 2 17:33:24 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4800116A4D0; Mon, 2 May 2005 17:33:24 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0947743D6A; Mon, 2 May 2005 17:33:23 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42HXIpe019123; Mon, 2 May 2005 19:33:18 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: "Steven Hartland" From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 17:43:16 BST." <003901c54f36$0c64ad40$b3db87d4@multiplay.co.uk> Date: Mon, 02 May 2005 19:33:18 +0200 Message-ID: <19122.1115055198@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Robert Watson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 17:33:24 -0000 In message <003901c54f36$0c64ad40$b3db87d4@multiplay.co.uk>, "Steven Hartland" writes: >Ok from what your saying it sounds like RAID on FreeBSD is useless >apart to create large disks. Now to the damaging facts the results >from my two days worth of testing: Now, cool down a moment and lets talk about what you _really_ have measured. You have measured the end to end performance: you include /dev/zero, dd(1), filesystem, disk device driver, hardware and disks. As such that is a fair end-user benchmark, but unfortunately it doesn't really tell us anything useful for the purpose of this discussion. If you are going to do a high-performance setup, you should not just take the "out-of-box" settings, you should optimize for the configurations particular features. Testing end-to-end means that we have very little to go from to find out where things went wrong in any one instance. But anyway: here are some questions which I wonder about ? Does the linux and FreeBSD filesystem offer the same semantics with respect to integrity ? Ie: if one is asyncronous mode the comparison is not a fair comparison. Just because you chose the default in each case doesn't mean you got the same thing. Does any of the drivers change the settings/modes of the controllers use of cache ? (this may be hard to determine without looking at driver sources. In particular, are you sure that the RAID-5 logical device was in the same exact state and location for each run ? Did you remember to disable all the debugging in FreeBSD 6-Current ? (see top of src/UPDATING) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 18:27:06 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 12E9216A4CE; Mon, 2 May 2005 18:27:06 +0000 (GMT) Received: from gate.bitblocks.com (bitblocks.com [209.204.185.216]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8154543D70; Mon, 2 May 2005 18:27:05 +0000 (GMT) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost [127.0.0.1]) by gate.bitblocks.com (8.13.3/8.13.1) with ESMTP id j42IR3KK007482; Mon, 2 May 2005 11:27:03 -0700 (PDT) (envelope-from bakul@bitblocks.com) Message-Id: <200505021827.j42IR3KK007482@gate.bitblocks.com> To: "Poul-Henning Kamp" In-reply-to: Your message of "Mon, 02 May 2005 19:33:18 +0200." <19122.1115055198@critter.freebsd.dk> Date: Mon, 02 May 2005 11:27:03 -0700 From: Bakul Shah cc: Robert Watson cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Steven Hartland Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 18:27:06 -0000 It may make sense to look at measured bandwidth as a percentage of *guaranteed not to exceed* bandwidth of the disk setup -- what is the theoretical max bandwidth writing to a raw partition (and assuming zero cpu overhead, latency, seek time)? This will help in figuring out how to maximize end-to-end performance, taking into account filesystem overhead, data integrity tradeoffs etc. One has to measure each component to find and fix the top N bottlenecks. This sort of critical analysis is what allowed people to make TCP/IP blazingly fast. For instance, Hartland reported max read/write performance of about 260M/s. *If* his disks can do, say, 80MB/s, for a 5 disk RAID5 he'd get 320MB/s. If so, his end-to-end measured performance would be over 80% of max (for whatever data integrity guarantees). Not too shabby:-) Improving end-to-end disk/os/filesystem performance can be quite an exciting (and time consuming) project. - do we have hooks to measure performance of each component? - do we have tests for these? - do we know the top 3 bottlenecks? - do we have a framework for independently experimenting with each component? (geom definitely helps here!) - do we have a framework for experimenting with various filesystem layout schemes? with various caching strategies? with various allocation strategies? - tools for locating bottlenecks due to mistuning? - tools to help tune the system? - how about tools to auto-tune the system?:) I agree with the questions you raised, but it seems we need to move the discussion to a meta level! I fully realize that in a volunteer project a) there are never enough people b) people works on what they like.... May be someone like you can inspire & put together a team to make freebsd the best damn storage OS in the world! Actually what would be really neat is to factor out all FS code across all *BSDs so that rather than fixing the same old bugs N times, people work on different FS designs. The reality is that open OSes are still nowhere near what SGI boxes of a few years ago could do. I have heard of some NAS box vendors achieving really good performance but that doesn't help us in the open OS community (not to mention their advances will be lost when the company dies or is garbage collected by the likes of EMC, netApp). From owner-freebsd-performance@FreeBSD.ORG Mon May 2 19:10:15 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ADDFF16A4CE; Mon, 2 May 2005 19:10:15 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9441A43D66; Mon, 2 May 2005 19:10:14 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001376250.msg; Mon, 02 May 2005 20:06:09 +0100 Message-ID: <005401c54f4a$7f271890$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Poul-Henning Kamp" References: <19122.1115055198@critter.freebsd.dk> Date: Mon, 2 May 2005 20:09:39 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 20:06:09 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 20:06:11 +0100 cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Robert Watson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 19:10:15 -0000 ----- Original Message ----- From: "Poul-Henning Kamp" >>Ok from what your saying it sounds like RAID on FreeBSD is useless >>apart to create large disks. Now to the damaging facts the results >>from my two days worth of testing: > > Now, cool down a moment and lets talk about what you _really_ have > measured. > > You have measured the end to end performance: you include /dev/zero, > dd(1), filesystem, disk device driver, hardware and disks. > > As such that is a fair end-user benchmark, but unfortunately it > doesn't really tell us anything useful for the purpose of this > discussion. Yes but the end-user performance is really the only thing that matters. There are two killer issues here: 1. Write performance being nearly 3x that of read performance 2. Read performance only equalling that of single disk. > If you are going to do a high-performance setup, you should not > just take the "out-of-box" settings, you should optimize for > the configurations particular features. I'm quite willing to test and optimise things but so far no one has had any concrete suggestions on that to try. As such I've done all the benchmarks I've done varying the one key thing I could find stripe size. If there are others please do tell and I'll test. N.B. I need to get this machine into production so I only have a limited time span available, I've already sacrificed a full install with 700GB of data to run the previous tests. This is just me though I think we do need to strive for good out the box performance in these types of senarios > Testing end-to-end means that we have very little to go from to > find out where things went wrong in any one instance. To eliminate various parts of the subsystems I've just tested: dd if=/dev/da0 of=/dev/null bs=64k count=100000 Read: 220Mb/s dd if=/dev/da0s1 of=/dev/null bs=64k count=100000 Read: 220Mb/s dd if=/dev/da0s1f of=/dev/null bs=64k count=100000 Read: 220Mb/s Compared with: dd if=/usr/testfile of=/dev/null bs=64k count=100000 Read: 152Mb/s So looks like the FS is adding quite an overhead ~70Mb/s ( 60% ) although from the linux tests we know the disks are capable of at least another 40Mb/s > But anyway: here are some questions which I wonder about ? > Does the linux and FreeBSD filesystem offer the same semantics > with respect to integrity ? Ie: if one is asyncronous mode > the comparison is not a fair comparison. Just because you > chose the default in each case doesn't mean you got the same > thing. FreeBSD using default UFS no special options in fstab. Linux using Reiser again no special options. I'm not familiar with the intrecuies of either FS so cant answer the direct question here. > Does any of the drivers change the settings/modes of the controllers > use of cache ? (this may be hard to determine without looking > at driver sources. FreeBSD used both the downloadable module and the built in module. Linux used the downloadable module. If these vary in any way I dont know may be Paul could answer that as he did the driver port. > In particular, are you sure that the RAID-5 logical device was > in the same exact state and location for each run ? Yep I created the RAID5 logical volume the same way via the BIOS for each test the only varient being the stripe size. > Did you remember to disable all the debugging in FreeBSD 6-Current ? > (see top of src/UPDATING) Yep all debugging was disabled on my second run on current. Made 3MB/s difference IIRC ( reported value was without debugging ). FreeBSD was tried with and without SMP no difference Linux was SMP for all tests. N.B. Current had at least on out of order lock issue while I was using it but not while the tests where going on. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 19:20:53 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 54AD316A4CE; Mon, 2 May 2005 19:20:53 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1548843D2D; Mon, 2 May 2005 19:20:52 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42JKmnH019880; Mon, 2 May 2005 21:20:48 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: "Steven Hartland" From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 20:09:39 BST." <005401c54f4a$7f271890$b3db87d4@multiplay.co.uk> Date: Mon, 02 May 2005 21:20:48 +0200 Message-ID: <19879.1115061648@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Robert Watson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 19:20:53 -0000 In message <005401c54f4a$7f271890$b3db87d4@multiplay.co.uk>, "Steven Hartland" writes: >> As such that is a fair end-user benchmark, but unfortunately it >> doesn't really tell us anything useful for the purpose of this >> discussion. > >Yes but the end-user performance is really the only thing that matters. >There are two killer issues here: No, there is three issues here, and you correctly identify the secondary two, but forget the first: 0. Does the user know enough about what he is doing. >1. Write performance being nearly 3x that of read performance >2. Read performance only equalling that of single disk. If the user expects an out of the box configuration with default parameters to give him maximal performance, the answer to issue number zero is: Obviously not. >I'm quite willing to test and optimise things but so far no one has >had any concrete suggestions on that to try. First thing I heard about this was a few hours ago. (Admittedly, my email has been in a sucky state last week, so that is probably my own fault). >This is just me though I think we do need to strive for good out >the box performance in these types of senarios We strive for a sensibly balanced system, no matter what use people put an out-of-the-box configuration to. >> Testing end-to-end means that we have very little to go from to >> find out where things went wrong in any one instance. > >To eliminate various parts of the subsystems I've just tested: >dd if=/dev/da0 of=/dev/null bs=64k count=100000 >Read: 220Mb/s This is a very interesting number to measure, you'll never see anything else going faster than that. Presumably this is -current ? >Compared with: >dd if=/usr/testfile of=/dev/null bs=64k count=100000 >Read: 152Mb/s On -current and 5.4 you don't have to make partitions if you intend to use the entire disk (and provided you don't want to boot from it). You can simply: newfs /dev/da0 mount /dev/da0 /where_ever This should have the sideeffect of aligning your filesystem correctly to the RAID volume. >So looks like the FS is adding quite an overhead ~70Mb/s ( 60% ) >although from the linux tests we know the disks are capable >of at least another 40Mb/s Yes, filesystems add overhead. That's just the way things are. One thing you could try is to use a larger block/fragment size on your filesystem. Try: newfs -b 32768 -f 4096 /dev/da0 >> Did you remember to disable all the debugging in FreeBSD 6-Current ? >> (see top of src/UPDATING) > >Yep all debugging was disabled on my second run on current. Just checking: what exactly did you disable ? >N.B. Current had at least on out of order lock issue while I was using >it but not while the tests where going on. Yes, current is current :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 20:44:34 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5A3B216A4CE; Mon, 2 May 2005 20:44:34 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2A27643D4C; Mon, 2 May 2005 20:44:33 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001376429.msg; Mon, 02 May 2005 21:40:30 +0100 Message-ID: <004101c54f57$abc04220$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Poul-Henning Kamp" References: <19879.1115061648@critter.freebsd.dk> Date: Mon, 2 May 2005 21:43:57 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 21:40:30 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 21:40:31 +0100 cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Robert Watson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 20:44:34 -0000 ----- Original Message ----- From: "Poul-Henning Kamp" > 0. Does the user know enough about what he is doing. Im no expert but then again Im not beginner either :) >>1. Write performance being nearly 3x that of read performance >>2. Read performance only equalling that of single disk. > > If the user expects an out of the box configuration with > default parameters to give him maximal performance, the > answer to issue number zero is: Obviously not. Not maximal, that will never be the case, but reasonable yes and it doesnt look like we are getting that ATM. >>I'm quite willing to test and optimise things but so far no one has >>had any concrete suggestions on that to try. > > First thing I heard about this was a few hours ago. (Admittedly, > my email has been in a sucky state last week, so that is probably > my own fault). Hehe its been kicking around for a few days, might be of benifit it you check back some some of the info posted by others. >>This is just me though I think we do need to strive for good out >>the box performance in these types of senarios > > We strive for a sensibly balanced system, no matter what use > people put an out-of-the-box configuration to. Indeed and I would classify this one of those such senarios. >>> Testing end-to-end means that we have very little to go from to >>> find out where things went wrong in any one instance. >> >>To eliminate various parts of the subsystems I've just tested: >>dd if=/dev/da0 of=/dev/null bs=64k count=100000 >>Read: 220Mb/s > > This is a very interesting number to measure, you'll never > see anything else going faster than that. Presumably > this is -current ? Nope thats 5.4-STABLE this should be at the very least 260Mb/s as thats what the controller has been measured on linux at even through the FS. >>Compared with: >>dd if=/usr/testfile of=/dev/null bs=64k count=100000 >>Read: 152Mb/s > > On -current and 5.4 you don't have to make partitions if you > intend to use the entire disk (and provided you don't want > to boot from it). You can simply: > > newfs /dev/da0 > mount /dev/da0 /where_ever Booting from it unfortunately. Although wasnt when running the big set of tests results I reported ealier. Gonna rip the machine out and put the test OS disk back in and try that. > This should have the sideeffect of aligning your filesystem > correctly to the RAID volume. > >>So looks like the FS is adding quite an overhead ~70Mb/s ( 60% ) >>although from the linux tests we know the disks are capable >>of at least another 40Mb/s > > Yes, filesystems add overhead. That's just the way things are. To be expected but 60% overhead is a bit excessive. > One thing you could try is to use a larger block/fragment size > on your filesystem. Try: > > newfs -b 32768 -f 4096 /dev/da0 Will do. >>> Did you remember to disable all the debugging in FreeBSD 6-Current ? >>> (see top of src/UPDATING) >> >>Yep all debugging was disabled on my second run on current. > > Just checking: what exactly did you disable ? Dont have it to hand any more witness etc, all the debugger options + stripped out all the unneeded drivers. >>N.B. Current had at least on out of order lock issue while I was using >>it but not while the tests where going on. > > Yes, current is current :-) As I thought hence its not running that any more bit to unstable for my main dev box :) Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 20:45:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 33F8D16A4D3 for ; Mon, 2 May 2005 20:45:10 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id BC76743D1D for ; Mon, 2 May 2005 20:45:09 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 3C7D446B45; Mon, 2 May 2005 16:45:09 -0400 (EDT) Date: Mon, 2 May 2005 21:48:19 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Poul-Henning Kamp In-Reply-To: <19879.1115061648@critter.freebsd.dk> Message-ID: <20050502214208.M87351@fledge.watson.org> References: <19879.1115061648@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Steven Hartland Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 20:45:10 -0000 On Mon, 2 May 2005, Poul-Henning Kamp wrote: >> I'm quite willing to test and optimise things but so far no one has >> had any concrete suggestions on that to try. > > First thing I heard about this was a few hours ago. (Admittedly, my > email has been in a sucky state last week, so that is probably my own > fault). This is because I resent a message on the -performance list to you and Scott Long, and Soren because I was concerned that the thread was identifying a reproduceable problem for several people, but no one had really bit in to try and diagnose it :-). It seems likely we're hurting at a couple of levels, and it would be good to identify what's appearing at what level. I think comparing dd results on the total disk device directly as a starting point is a good place to begin. I'm not sure if we've seen Linux and FreeBSD dmsg output yet, but if nothing else it would be good to confirm if the drivers on both systems negotiate the same level of throughput to each drive. The next thing that would be quite nice to measure is the rate of I/O transactions per second we can get to the disk using the disk device directly, with a minimal transaction size. I have a vague recollection that you have to be careful in Linux because their character device nodes for disk devices are buffered, and you really want unbuffered I/O. After transactions/sec, broad sweeps across well-understood areas of the disk would be a good thing to do, again using the device node directly. Isolating the impact of the file system will be important... Robert N M Watson From owner-freebsd-performance@FreeBSD.ORG Mon May 2 20:54:03 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 177E816A4CE for ; Mon, 2 May 2005 20:54:03 +0000 (GMT) Received: from ford.blinkenlights.nl (ford.blinkenlights.nl [213.204.211.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id B9D8043D53 for ; Mon, 2 May 2005 20:54:02 +0000 (GMT) (envelope-from sten@blinkenlights.nl) Received: from tea.blinkenlights.nl (tea.blinkenlights.nl [192.168.1.21]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ford.blinkenlights.nl (Postfix) with ESMTP id F23703F294; Mon, 2 May 2005 22:54:00 +0200 (CEST) Received: by tea.blinkenlights.nl (Postfix, from userid 101) id 8EFBA272; Mon, 2 May 2005 22:54:00 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by tea.blinkenlights.nl (Postfix) with ESMTP id 832CD79; Mon, 2 May 2005 22:54:00 +0200 (CEST) Date: Mon, 2 May 2005 22:54:00 +0200 (CEST) From: Sten Spans To: Poul-Henning Kamp In-Reply-To: <18194.1115044656@critter.freebsd.dk> Message-ID: References: <18194.1115044656@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: Steven Hartland cc: freebsd-performance@freebsd.org cc: Allen cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 20:54:03 -0000 On Mon, 2 May 2005, Poul-Henning Kamp wrote: > In message <42763906.1040202@centtech.com>, Eric Anderson writes: >>> In message <6.2.1.2.2.20050502094757.037077f0@mail.rfnj.org>, Allen writes: >>> >>> I just want to add: This is why I really would love for us to have >>> a real RAID3 implemetation. >>> >>> RAID3 is not commercially viable because windows cannot use non-512 >>> byte sectors. >>> >>> We can. >>> >>> RAID3 would scream for us. >> >> What about disk arrays that support RAID3? > > Would work for me, but most of them are dumbed down when they do RAID3: > they have to hard format the disks to 128 byte sector sizes and similar > madness in order to support 512 bytes sectors on the RAID3 volume. I would really love the 512 + 8 byte checksum stuff that mainframes and netapps do. Does GEOM simplify implementing something like this ? -- Sten Spans "There is a crack in everything, that's how the light gets in." Leonard Cohen - Anthem From owner-freebsd-performance@FreeBSD.ORG Mon May 2 20:58:09 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6EAA716A4F1 for ; Mon, 2 May 2005 20:58:09 +0000 (GMT) Received: from lakepoint.domeneshop.no (lakepoint.domeneshop.no [194.63.248.54]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8777843D82 for ; Mon, 2 May 2005 20:58:08 +0000 (GMT) (envelope-from lists@wm-access.no) Received: from [192.168.1.12] (host-81-191-1-177.bluecom.no [81.191.1.177]) (authenticated bits=0)j42Kw5Ox016343; Mon, 2 May 2005 22:58:06 +0200 Message-ID: <42763F28.7020502@wm-access.no> Date: Mon, 02 May 2005 16:54:32 +0200 From: =?ISO-8859-1?Q?Sten_Daniel_S=F8rsdal?= User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Mike Tancsa References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> In-Reply-To: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 20:58:09 -0000 Mike Tancsa wrote: > A somewhat obvious question to some perhaps, but what server application > mix on FreeBSD today sees an improvement using 64bit CPUs ? In my ISP > centric world, my big apps are BIND, IMAP/POP3, httpd via apache, SMTP, > AV and SPAM scanning, and firewalls/routing. Apart from larger RAM, why > would these benefit from the 64bit world ? Or would they ? Any application that would benefit from being able to copy 64 bits at a time, i assume. Video applications and perhaps some encryption applications. Perhaps IPv6 core would have a slight benefit (although probably not with current code). Almost all the applications that you mention are doing mostly string operations (afaik). Perhaps VPN tunnels could benefit (unless of course you use hardware vpns like we do). -- Sten Daniel Sørsdal From owner-freebsd-performance@FreeBSD.ORG Mon May 2 21:05:05 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 08D5516A4CF for ; Mon, 2 May 2005 21:05:05 +0000 (GMT) Received: from mail.rfnj.org (ns1.rfnj.org [66.180.172.156]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5045A43D81 for ; Mon, 2 May 2005 21:05:04 +0000 (GMT) (envelope-from asym@rfnj.org) Received: from megalomaniac.rfnj.org (ool-45736df1.dyn.optonline.net [69.115.109.241]) by mail.rfnj.org (Postfix) with ESMTP id 740F7173; Mon, 2 May 2005 17:05:04 -0400 (EDT) Message-Id: <6.2.1.2.2.20050502170756.037186b0@mail.rfnj.org> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Mon, 02 May 2005 17:11:02 -0400 To: Sten Daniel =?iso-8859-1?Q?S=F8rsdal?= , Mike Tancsa From: Allen In-Reply-To: <42763F28.7020502@wm-access.no> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <42763F28.7020502@wm-access.no> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable cc: freebsd-performance@freebsd.org Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 21:05:05 -0000 At 10:54 5/2/2005, Sten Daniel S=F8rsdal wrote: >Mike Tancsa wrote: > > A somewhat obvious question to some perhaps, but what server application > > mix on FreeBSD today sees an improvement using 64bit CPUs ? In my ISP > > centric world, my big apps are BIND, IMAP/POP3, httpd via apache, SMTP, > > AV and SPAM scanning, and firewalls/routing. Apart from larger RAM, why > > would these benefit from the 64bit world ? Or would they ? > >Any application that would benefit from being able to copy 64 bits at a >time, i assume. Video applications and perhaps some encryption >applications. Perhaps IPv6 core would have a slight benefit (although >probably not with current code). Almost all the applications that you >mention are doing mostly string operations (afaik). Perhaps VPN tunnels >could benefit (unless of course you use hardware vpns like we do). The real benefit for ISP servers comes in from the 64bit address space, not= =20 the 64bit registers, though they can help out occasionally in those=20 workloads as you mentioned. Having 1MB+ of L2 cache, and supporting 20+GB of memory are the real wins=20 in your case. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 21:06:56 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A8EEC16A4CE for ; Mon, 2 May 2005 21:06:56 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id A013143D67 for ; Mon, 2 May 2005 21:06:55 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j42L6nnS020785; Mon, 2 May 2005 23:06:49 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Sten Spans From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 02 May 2005 22:54:00 +0200." Date: Mon, 02 May 2005 23:06:49 +0200 Message-ID: <20784.1115068009@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: Steven Hartland cc: freebsd-performance@freebsd.org cc: Allen cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 21:06:56 -0000 In message , Sten Spans w rites: >>> What about disk arrays that support RAID3? >> >> Would work for me, but most of them are dumbed down when they do RAID3: >> they have to hard format the disks to 128 byte sector sizes and similar >> madness in order to support 512 bytes sectors on the RAID3 volume. > >I would really love the 512 + 8 byte checksum stuff that mainframes >and netapps do. Does GEOM simplify implementing something like this ? Yes, GEOM works with arbitrary sectorsizes, but far from all current GEOM classes does. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 21:16:53 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7A8C416A4D2; Mon, 2 May 2005 21:16:53 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id A0F8B43D1F; Mon, 2 May 2005 21:16:52 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001376499.msg; Mon, 02 May 2005 22:13:14 +0100 Message-ID: <008201c54f5c$3f075d80$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Robert Watson" , "Poul-Henning Kamp" References: <19879.1115061648@critter.freebsd.dk> <20050502214208.M87351@fledge.watson.org> Date: Mon, 2 May 2005 22:16:42 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 22:13:14 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 22:13:15 +0100 cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 21:16:53 -0000 ----- Original Message ----- From: "Robert Watson" > I'm not sure if we've seen Linux and FreeBSD dmsg output yet, but > if nothing else it would be good to confirm if the drivers on both systems > negotiate the same level of throughput to each drive. Both drivers ( FreeBSD and Linux ) have identical output on startup. May 3 22:12:26 r2d2 kernel: hptmv0: mem 0xf8200000-0xf827ffff irq 25 at device 1.0 on pci9 May 3 22:12:26 r2d2 kernel: RocketRAID 182x SATA Controller driver Version 1.1 May 3 22:12:26 r2d2 kernel: RR182x [0,0]: channel started successfully May 3 22:12:26 r2d2 kernel: RR182x [0,1]: channel started successfully May 3 22:12:26 r2d2 kernel: RR182x [0,2]: channel started successfully May 3 22:12:26 r2d2 kernel: RR182x [0,3]: channel started successfully May 3 22:12:26 r2d2 kernel: RR182x [0,4]: channel started successfully May 3 22:12:26 r2d2 kernel: RR182x: RAID5 write-back enabled ... May 3 22:12:26 r2d2 kernel: da0 at hptmv0 bus 0 target 0 lun 0 May 3 22:12:26 r2d2 kernel: da0: Fixed Direct Access SCSI-0 device May 3 22:12:26 r2d2 kernel: da0: 1526216MB (3125691008 512 byte sectors: 255H 63S/T 194565C) I've now got the noisy beast next to me here just installing 5.4-STABLE on my test OS disk then free to do what ever tests u guys need doing. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 21:17:56 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 04C2516A4CF for ; Mon, 2 May 2005 21:17:56 +0000 (GMT) Received: from smtp819.mail.sc5.yahoo.com (smtp819.mail.sc5.yahoo.com [66.163.170.5]) by mx1.FreeBSD.org (Postfix) with SMTP id 58E1043D1F for ; Mon, 2 May 2005 21:17:55 +0000 (GMT) (envelope-from noackjr@alumni.rice.edu) Received: from unknown (HELO optimator.noacks.org) (noacks@swbell.net@70.240.205.64 with login) by smtp819.mail.sc5.yahoo.com with SMTP; 2 May 2005 21:17:54 -0000 Received: from localhost (localhost [127.0.0.1]) by optimator.noacks.org (Postfix) with ESMTP id EE2D1610A; Mon, 2 May 2005 16:17:53 -0500 (CDT) Received: from optimator.noacks.org ([127.0.0.1]) by localhost (optimator.noacks.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 07761-12-4; Mon, 2 May 2005 16:17:51 -0500 (CDT) Received: from [127.0.0.1] (optimator [192.168.1.11]) by optimator.noacks.org (Postfix) with ESMTP id 36E8C60D2; Mon, 2 May 2005 16:17:51 -0500 (CDT) Message-ID: <427698F9.8030501@alumni.rice.edu> Date: Mon, 02 May 2005 16:17:45 -0500 From: Jonathan Noack User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Steven Hartland References: <19879.1115061648@critter.freebsd.dk> <004101c54f57$abc04220$b3db87d4@multiplay.co.uk> In-Reply-To: <004101c54f57$abc04220$b3db87d4@multiplay.co.uk> X-Enigmail-Version: 0.91.0.0 OpenPGP: id=991D8195; url=http://www.noacks.org/cert/noackjr.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigAF948B5FA8AE6B7B648DD275" X-Virus-Scanned: amavisd-new at noacks.org cc: Robert Watson cc: Poul-Henning Kamp cc: Eric Anderson cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: noackjr@alumni.rice.edu List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 21:17:56 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigAF948B5FA8AE6B7B648DD275 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 5/2/2005 3:43 PM, Steven Hartland wrote: > ----- Original Message ----- From: "Poul-Henning Kamp" >>> To eliminate various parts of the subsystems I've just tested: >>> dd if=/dev/da0 of=/dev/null bs=64k count=100000 >>> Read: 220Mb/s >> >> This is a very interesting number to measure, you'll never >> see anything else going faster than that. Presumably >> this is -current ? > > Nope thats 5.4-STABLE this should be at the very least > 260Mb/s as thats what the controller has been measured on > linux at even through the FS. Um... not quite. That was the number you listed for S/W RAID5. In that case you're not benchmarking the controller in the same way; the controller is just serving requests with no RAID processing overhead at all. Could you get results for Linux that bypassed the filesystem but used H/W RAID5 like you are with FreeBSD? >>> Compared with: >>> dd if=/usr/testfile of=/dev/null bs=64k count=100000 >>> Read: 152Mb/s -- Jonathan Noack | noackjr@alumni.rice.edu | OpenPGP: 0x991D8195 --------------enigAF948B5FA8AE6B7B648DD275 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (MingW32) iD8DBQFCdpj9UFz01pkdgZURAiy4AJ9gNWWnE/F4Wnui9HVk5c/UDZNYMwCgxQYx H1N2wcElnMeLOMyjAz7YYQs= =sXiJ -----END PGP SIGNATURE----- --------------enigAF948B5FA8AE6B7B648DD275-- From owner-freebsd-performance@FreeBSD.ORG Mon May 2 21:31:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C90BD16A4CE; Mon, 2 May 2005 21:31:10 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id D2CEE43D68; Mon, 2 May 2005 21:31:09 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001376525.msg; Mon, 02 May 2005 22:27:01 +0100 Message-ID: <00b601c54f5e$2bae23c0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: References: <19879.1115061648@critter.freebsd.dk><004101c54f57$abc04220$b3db87d4@multiplay.co.uk> <427698F9.8030501@alumni.rice.edu> Date: Mon, 2 May 2005 22:30:29 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 22:27:01 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 22:27:03 +0100 cc: Eric Anderson cc: Poul-Henning Kamp cc: Robert Watson cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 21:31:11 -0000 On 5/2/2005 3:43 PM, Steven Hartland wrote: >> Nope thats 5.4-STABLE this should be at the very least >> 260Mb/s as thats what the controller has been measured on >> linux at even through the FS. > >Um... not quite. That was the number you listed for S/W RAID5. In that >case you're not benchmarking the controller in the same way; the >controller is just serving requests with no RAID processing overhead at >all. Could you get results for Linux that bypassed the filesystem but >used H/W RAID5 like you are with FreeBSD? This is true would need to do the H/W RAID5 test on a raw volume to be sure that the H/W RAID is not a bottleneck. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 21:58:16 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D58BD16A4CE; Mon, 2 May 2005 21:58:16 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id E84A343D1F; Mon, 2 May 2005 21:58:15 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001376566.msg; Mon, 02 May 2005 22:53:32 +0100 Message-ID: <00fe01c54f61$df220d60$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Poul-Henning Kamp" References: <19879.1115061648@critter.freebsd.dk> Date: Mon, 2 May 2005 22:56:57 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 22:53:32 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 22:53:34 +0100 cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 21:58:17 -0000 ----- Original Message ----- From: "Poul-Henning Kamp" > On -current and 5.4 you don't have to make partitions if you > intend to use the entire disk (and provided you don't want > to boot from it). You can simply: > > newfs /dev/da0 > mount /dev/da0 /where_ever /dev/da0: 1526216.3MB (3125691008 sectors) block size 16384, fragment size 2048 using 8306 cylinder groups of 183.77MB, 11761 blks, 23552 inodes. mount /dev/da0 /mnt dd if=/dev/zero of=/mnt/testfile bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 44.424588 secs (147521908 bytes/sec) /usr/bin/time -h dd of=/dev/null if=/mnt/testfile bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 48.150473 secs (136106658 bytes/sec) 48.27s real 0.01s user 12.33s sys /usr/bin/time -h cat /mnt/testfile > /dev/null 48.12s real 0.26s user 14.92s sys Raw read: /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 32.028544 secs (204617482 bytes/sec) 32.02s real 0.02s user 2.73s sys /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=1m count=6000 6000+0 records in 6000+0 records out 6291456000 bytes transferred in 31.518195 secs (199613460 bytes/sec) 31.51s real 0.00s user 1.87s sys > One thing you could try is to use a larger block/fragment size > on your filesystem. Try: > > newfs -b 32768 -f 4096 /dev/da0 /dev/da0: 1526216.3MB (3125691008 sectors) block size 32768, fragment size 4096 using 2063 cylinder groups of 740.00MB, 23680 blks, 47360 inodes. /usr/bin/time -h dd if=/dev/zero of=/mnt/testfile bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 35.237328 secs (185984590 bytes/sec) 35.34s real 0.07s user 28.81s sys /usr/bin/time -h dd of=/dev/null if=/mnt/testfile bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 39.154845 secs (167376477 bytes/sec) 39.16s real 0.01s user 14.33s sys Slower than the default :( ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 22:19:11 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6B69E16A4D6 for ; Mon, 2 May 2005 22:19:11 +0000 (GMT) Received: from web41205.mail.yahoo.com (web41205.mail.yahoo.com [66.218.93.38]) by mx1.FreeBSD.org (Postfix) with SMTP id D198243D68 for ; Mon, 2 May 2005 22:19:10 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 36687 invoked by uid 60001); 2 May 2005 22:19:10 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=aV35ZJ/F8peD2d2jbr3aFmu6PKJWkYJHkzIQ9zLzTGqfiO8sXd/U+prhxoDEYZxwNLSvfT/Zs5bL4l2QQywcRzTlBhTG4uejwAWNdscUgy+HH1KRMo3tP2VojrjtRyWZoRJjCQTcMiMm1SRCH0h1SOxLDDlLFA/C78H65fyMzhI= ; Message-ID: <20050502221910.36685.qmail@web41205.mail.yahoo.com> Received: from [213.54.148.161] by web41205.mail.yahoo.com via HTTP; Mon, 02 May 2005 15:19:10 PDT Date: Mon, 2 May 2005 15:19:10 -0700 (PDT) From: Arne "Wörner" To: Steven Hartland , Poul-Henning Kamp In-Reply-To: <00fe01c54f61$df220d60$b3db87d4@multiplay.co.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Robert Watson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 22:19:11 -0000 --- Steven Hartland wrote: > ----- Original Message ----- > From: "Poul-Henning Kamp" > > On -current and 5.4 you don't have to make partitions if you > > intend to use the entire disk (and provided you don't want > > to boot from it). You can simply: > > > > newfs /dev/da0 > > mount /dev/da0 /where_ever > > /dev/da0: 1526216.3MB (3125691008 sectors) block size 16384, > fragment size 2048 > using 8306 cylinder groups of 183.77MB, 11761 blks, 23552 > inodes. > mount /dev/da0 /mnt > > dd if=/dev/zero of=/mnt/testfile bs=64k count=100000 > 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 44.424588 secs (147521908 > bytes/sec) > > /usr/bin/time -h dd of=/dev/null if=/mnt/testfile bs=64k > count=100000 > 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 48.150473 secs (136106658 > bytes/sec) > 48.27s real 0.01s user 12.33s > sys > > /usr/bin/time -h cat /mnt/testfile > /dev/null > 48.12s real 0.26s user 14.92s > sys > > Raw read: > /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k count=100000 > > 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 32.028544 secs (204617482 > bytes/sec) > 32.02s real 0.02s user 2.73s > sys > > /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=1m count=6000 > 6000+0 records in > 6000+0 records out > 6291456000 bytes transferred in 31.518195 secs (199613460 > bytes/sec) > 31.51s real 0.00s user 1.87s > sys > > > One thing you could try is to use a larger block/fragment size > > on your filesystem. Try: > > > > newfs -b 32768 -f 4096 /dev/da0 > /dev/da0: 1526216.3MB (3125691008 sectors) block size 32768, > fragment size 4096 > using 2063 cylinder groups of 740.00MB, 23680 blks, > 47360 inodes. > > /usr/bin/time -h dd if=/dev/zero of=/mnt/testfile bs=64k > count=100000 > 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 35.237328 secs (185984590 > bytes/sec) > 35.34s real 0.07s user 28.81s > sys > > /usr/bin/time -h dd of=/dev/null if=/mnt/testfile bs=64k > count=100000 > 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 39.154845 secs (167376477 > bytes/sec) > 39.16s real 0.01s user 14.33s > sys > > Slower than the default :( > Why? It looks like the 32k-bs-fs terminates 20% earlier than the 16k-bs-fs. Furthermore I would like to re-interpret my analysis about my hard disc /dev/ad0: It looks like, I damaged the write cache somehow... atacontrol's output is not consistent in this point (it changes from reboot to reboot, while the write rate is always 5 times lower than the read rate), while hw.ata.wc is always 1. So I just see, that there is some overhead. Example: my /dev/ad1: 92MB/sec theoretical maximum (data transfer rate from the medium (disc) to device's buffer) 60MB/sec if I read bs=128k 40MB/sec if I read from a ufs+s with std bs (16kB?) -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Mon May 2 22:23:14 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EA8A116A4CE; Mon, 2 May 2005 22:23:13 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id DCE0143D7F; Mon, 2 May 2005 22:23:12 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001376600.msg; Mon, 02 May 2005 23:19:28 +0100 Message-ID: <001f01c54f65$80231080$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Steven Hartland" , "Poul-Henning Kamp" References: <19879.1115061648@critter.freebsd.dk> <00fe01c54f61$df220d60$b3db87d4@multiplay.co.uk> Date: Mon, 2 May 2005 23:22:57 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Mon, 02 May 2005 23:19:28 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Mon, 02 May 2005 23:19:30 +0100 cc: Eric Anderson cc: freebsd-performance@freebsd.org cc: Robert Watson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 22:23:14 -0000 ----- Original Message ----- > > Raw read: > /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k count=100000 > 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 32.028544 secs (204617482 bytes/sec) > 32.02s real 0.02s user 2.73s sys Out of curiosity just recreated the array with a 64K stripe ( the default ) /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 61.246992 secs (107002806 bytes/sec) 1m1.24s real 0.06s user 2.83s sys /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=128k count=50000 50000+0 records in 50000+0 records out 6553600000 bytes transferred in 59.532993 secs (110083496 bytes/sec) 59.53s real 0.00s user 1.86s sys And a 32k stripe: /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 41.947912 secs (156231853 bytes/sec) 41.94s real 0.04s user 2.80s sys So here the smaller the stripe the higher the transfer rate. Some RAID0 results: 64k stripe: /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 59.255960 secs (110598158 bytes/sec) 59.25s real 0.04s user 2.52s sys 32k stripe: /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 39.402059 secs (166326333 bytes/sec) 39.40s real 0.00s user 2.96s sys 16k stripe: /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 31.954544 secs (205091332 bytes/sec) 31.95s real 0.04s user 2.73s sys Conclusion from this is that at 16K stripe size RAID5 is identical read performance of RAID0 which is what I would expect. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Mon May 2 22:28:57 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B30E016A4D0 for ; Mon, 2 May 2005 22:28:57 +0000 (GMT) Received: from smtp818.mail.sc5.yahoo.com (smtp818.mail.sc5.yahoo.com [66.163.170.4]) by mx1.FreeBSD.org (Postfix) with SMTP id EE28243D6E for ; Mon, 2 May 2005 22:28:56 +0000 (GMT) (envelope-from noackjr@alumni.rice.edu) Received: from unknown (HELO optimator.noacks.org) (noacks@swbell.net@70.240.205.64 with login) by smtp818.mail.sc5.yahoo.com with SMTP; 2 May 2005 22:16:55 -0000 Received: from localhost (localhost [127.0.0.1]) by optimator.noacks.org (Postfix) with ESMTP id 5BED7610A; Mon, 2 May 2005 17:16:54 -0500 (CDT) Received: from optimator.noacks.org ([127.0.0.1]) by localhost (optimator.noacks.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 08669-04; Mon, 2 May 2005 17:16:50 -0500 (CDT) Received: from [127.0.0.1] (optimator [192.168.1.11]) by optimator.noacks.org (Postfix) with ESMTP id 3DF0960D2; Mon, 2 May 2005 17:16:50 -0500 (CDT) Message-ID: <4276A6CC.3000108@alumni.rice.edu> Date: Mon, 02 May 2005 17:16:44 -0500 From: Jonathan Noack User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Steven Hartland References: <19879.1115061648@critter.freebsd.dk> <00fe01c54f61$df220d60$b3db87d4@multiplay.co.uk> In-Reply-To: <00fe01c54f61$df220d60$b3db87d4@multiplay.co.uk> X-Enigmail-Version: 0.91.0.0 OpenPGP: id=991D8195; url=http://www.noacks.org/cert/noackjr.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigDA032C7B1FAEFCCE124B0366" X-Virus-Scanned: amavisd-new at noacks.org cc: Eric Anderson cc: Poul-Henning Kamp cc: Robert Watson cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: noackjr@alumni.rice.edu List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 22:28:57 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigDA032C7B1FAEFCCE124B0366 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 5/2/2005 4:56 PM, Steven Hartland wrote: > ----- Original Message ----- From: "Poul-Henning Kamp" >> On -current and 5.4 you don't have to make partitions if you >> intend to use the entire disk (and provided you don't want >> to boot from it). You can simply: >> >> newfs /dev/da0 >> mount /dev/da0 /where_ever > > /dev/da0: 1526216.3MB (3125691008 sectors) block size 16384, fragment > size 2048 > using 8306 cylinder groups of 183.77MB, 11761 blks, 23552 inodes. > mount /dev/da0 /mnt > > dd if=/dev/zero of=/mnt/testfile bs=64k count=100000 > 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 44.424588 secs (147521908 bytes/sec) > > /usr/bin/time -h dd of=/dev/null if=/mnt/testfile bs=64k > count=100000 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 48.150473 secs (136106658 bytes/sec) > 48.27s real 0.01s user 12.33s sys > > /usr/bin/time -h cat /mnt/testfile > /dev/null > 48.12s real 0.26s user 14.92s sys > > Raw read: > /usr/bin/time -h dd of=/dev/null if=/dev/da0 bs=64k > count=100000 100000+0 records in > 100000+0 records out > 6553600000 bytes transferred in 32.028544 secs (204617482 bytes/sec) > 32.02s real 0.02s user 2.73s sys Look at the difference in sys times for raw vs. filesystem reads. With raw we're at 2.73s while reading from the filesystem requires 12.33s! From my position of complete ignorance that seems like a lot... -- Jonathan Noack | noackjr@alumni.rice.edu | OpenPGP: 0x991D8195 --------------enigDA032C7B1FAEFCCE124B0366 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (MingW32) iD8DBQFCdqbQUFz01pkdgZURAs7XAJ4oZksFCPBuH65I30UylE+HZhCJHgCeOXpw E6YXzImxHepv1kqpyHPwDz8= =XBU1 -----END PGP SIGNATURE----- --------------enigDA032C7B1FAEFCCE124B0366-- From owner-freebsd-performance@FreeBSD.ORG Mon May 2 23:03:48 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 721E916A4CE for ; Mon, 2 May 2005 23:03:48 +0000 (GMT) Received: from flake.decibel.org (flake.decibel.org [67.100.216.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 11BBD43D86 for ; Mon, 2 May 2005 23:03:48 +0000 (GMT) (envelope-from decibel@decibel.org) Received: by flake.decibel.org (Postfix, from userid 1001) id 7E0BF152B6; Mon, 2 May 2005 18:03:43 -0500 (CDT) Date: Mon, 2 May 2005 18:03:43 -0500 From: "Jim C. Nasby" To: David Gilbert Message-ID: <20050502230343.GG47820@decibel.org> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <42750861.8000509@mac.com> <6.2.1.2.0.20050501193659.05483bd0@64.7.153.2> <17014.14219.963494.753741@canoe.dclg.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <17014.14219.963494.753741@canoe.dclg.ca> X-Operating-System: FreeBSD 4.11-RELEASE i386 X-Distributed: Join the Effort! http://www.distributed.net User-Agent: Mutt/1.5.8i cc: freebsd-performance@freebsd.org cc: Mike Tancsa Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 23:03:48 -0000 On Mon, May 02, 2005 at 10:22:03AM -0400, David Gilbert wrote: > >>>>> "Mike" == Mike Tancsa writes: > > Mike> Thanks! I was worried I was missing something obvious in all > Mike> the 64bit excitement. For my apps where I cant balance across > Mike> multiple machines or where space is an issue, dual core CPUs I > Mike> think might be more interesting to look at in a few months. > > Hi Mike! > > Anyways, we've been finding that raw packet passing is significantly > better on some of the newer tyan opteron boards. But then that may > have as much to do with them shipping with multiple independant PCI-X > slots than the opterons themselves. Memory bandwidth probably doesn't > hurt, either. I was just going to mention that what's more important than gobs of memory is memory bandwidth, which the Opterons have much more of than other CPUs. So any application that needs to move a lot of data, whether a database or a router, will benefit. -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?" From owner-freebsd-performance@FreeBSD.ORG Mon May 2 23:07:08 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F107F16A4CE for ; Mon, 2 May 2005 23:07:08 +0000 (GMT) Received: from flake.decibel.org (flake.decibel.org [67.100.216.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id A8E7543D54 for ; Mon, 2 May 2005 23:07:08 +0000 (GMT) (envelope-from decibel@decibel.org) Received: by flake.decibel.org (Postfix, from userid 1001) id EBF92152B4; Mon, 2 May 2005 18:07:02 -0500 (CDT) Date: Mon, 2 May 2005 18:07:02 -0500 From: "Jim C. Nasby" To: Mike Tancsa Message-ID: <20050502230702.GH47820@decibel.org> References: <6.2.1.2.0.20050501094429.06974910@64.7.153.2> <20050501191001.GF85317@over-yonder.net> <6.2.1.2.0.20050501183331.045c3770@64.7.153.2> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6.2.1.2.0.20050501183331.045c3770@64.7.153.2> X-Operating-System: FreeBSD 4.11-RELEASE i386 X-Distributed: Join the Effort! http://www.distributed.net User-Agent: Mutt/1.5.8i cc: freebsd-performance@freebsd.org cc: "Matthew D. Fuller" Subject: Re: 64bit CPUs X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 23:07:09 -0000 On Sun, May 01, 2005 at 07:34:31PM -0400, Mike Tancsa wrote: > I havent paid that much attention to the comparisons as they tend to be > very windows based and games to boot. But surely its hard to compare > across platforms. The CPUs require different chipsets, so some of the > performance results can be due to the MB and RAM. > http://www6.tomshardware.com/cpu/20050221/prescott-10.html#synthetic > The P4 640 3.2Ghz is about the same price at one of my suppliers as the 64 > 3500+ and the numbers are not that different. > > For me to switch to a different platform and risk stability issues there > would have to be some decent payoff in either cost savings or serious > performance differences. I am thinking perhaps for my RADIUS accounting > database which is very large might benefit as I have already maxed out the > RAM. >From what I've seen on the PostgreSQL lists, PostgreSQL sees a huge (30%) performance increase on Opterons over Xeons, and other databases see 10-15%. I haven't seen 32 bit vs 64 bit numbers, but I would expect the increase to be even larger than Xeon to Opteron numbers. -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?" From owner-freebsd-performance@FreeBSD.ORG Mon May 2 23:38:00 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BCFCE16A4CE; Mon, 2 May 2005 23:38:00 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id E58EB43D49; Mon, 2 May 2005 23:37:59 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001376720.msg; Tue, 03 May 2005 00:33:37 +0100 Message-ID: <002c01c54f6f$da10deb0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: References: <19879.1115061648@critter.freebsd.dk> <00fe01c54f61$df220d60$b3db87d4@multiplay.co.uk> <4276A6CC.3000108@alumni.rice.edu> Date: Tue, 3 May 2005 00:36:59 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Tue, 03 May 2005 00:33:37 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Tue, 03 May 2005 00:33:39 +0100 cc: Eric Anderson cc: Poul-Henning Kamp cc: Robert Watson cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 May 2005 23:38:00 -0000 On 5/2/2005 4:56 PM, Jonathan Noack Look at the difference in sys times for raw vs. filesystem reads. With > raw we're at 2.73s while reading from the filesystem requires 12.33s! > From my position of complete ignorance that seems like a lot... Indeed thats why I hit on using time as well as just the dd stats. Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Tue May 3 04:37:24 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D2FC416A4CE; Tue, 3 May 2005 04:37:24 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1625F43D70; Tue, 3 May 2005 04:37:24 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [193.64.42.134] (h86.vuokselantie10.fi [193.64.42.134]) by silver.he.iki.fi (Postfix) with ESMTP id BF3ADBC46; Tue, 3 May 2005 07:37:21 +0300 (EEST) Message-ID: <42770026.80901@he.iki.fi> Date: Tue, 03 May 2005 07:37:58 +0300 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: <19879.1115061648@critter.freebsd.dk> <20050502214208.M87351@fledge.watson.org> In-Reply-To: <20050502214208.M87351@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: Steven Hartland cc: Poul-Henning Kamp cc: Eric Anderson cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 04:37:24 -0000 Robert Watson wrote: > > The next thing that would be quite nice to measure is the rate of I/O > transactions per second we can get to the disk using the disk device > directly, with a minimal transaction size. I have a vague > recollection that you have to be careful in Linux because their > character device nodes for disk devices are buffered, and you really > want unbuffered I/O. I noticed that changing vfs.read_max from the default 8 to 16 has a dramatic effect on sequential read performance. Increasing it further did not have measurable effect. Pete From owner-freebsd-performance@FreeBSD.ORG Tue May 3 06:45:54 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A373616A4CE; Tue, 3 May 2005 06:45:54 +0000 (GMT) Received: from mxfep01.bredband.com (mxfep01.bredband.com [195.54.107.70]) by mx1.FreeBSD.org (Postfix) with ESMTP id AC5D143D82; Tue, 3 May 2005 06:45:52 +0000 (GMT) (envelope-from martin@gneto.com) Received: from as6-1-5.kr.m.bonet.se ([83.227.181.30] [83.227.181.30]) by mxfep01.bredband.com with ESMTP <20050503064551.DYTG8970.mxfep01.bredband.com@as6-1-5.kr.m.bonet.se>; Tue, 3 May 2005 08:45:51 +0200 Received: from [192.168.10.11] (euklides.gneto.com [192.168.10.11]) by as6-1-5.kr.m.bonet.se (Postfix) with ESMTP id B4EAF67831; Tue, 3 May 2005 08:45:49 +0200 (CEST) Message-ID: <42771E1D.9070408@gneto.com> Date: Tue, 03 May 2005 08:45:49 +0200 From: Martin Nilsson User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050326) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Petri Helenius References: <19879.1115061648@critter.freebsd.dk> <20050502214208.M87351@fledge.watson.org> <42770026.80901@he.iki.fi> In-Reply-To: <42770026.80901@he.iki.fi> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: Steven Hartland cc: Poul-Henning Kamp cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 06:45:54 -0000 Petri Helenius wrote: > I noticed that changing vfs.read_max from the default 8 to 16 has a > dramatic effect on sequential read performance. Increasing it further > did not have measurable effect. Increasing MAXPHYS in sys/param.h from 128k to 1M increased sequential read thruput on my MegaRAID 1600 from 100MB/s to 190MB/s measured on 5.4 #RAID0 7*36GB MegaRAID 1600 MAXPHYS=1M t125# dd if=/dev/amrd0 of=/dev/null bs=1024k count=1024 1073741824 bytes transferred in 5.559722 secs (193128691 bytes/sec) It had no effect on bonnie so the filesystem probaby never issues block-reads that large, I'll try the above sysctl. 190MB/s is pretty good considering that the card is in a ServerWorks LE chipped P3 and that the internal datapath on the Elite 1600 is only 64-bit/33MHz. /Martin From owner-freebsd-performance@FreeBSD.ORG Tue May 3 07:06:18 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1118B16A4D1 for ; Tue, 3 May 2005 07:06:18 +0000 (GMT) Received: from smtp812.mail.sc5.yahoo.com (smtp812.mail.sc5.yahoo.com [66.163.170.82]) by mx1.FreeBSD.org (Postfix) with SMTP id 4E11243D6A for ; Tue, 3 May 2005 07:06:17 +0000 (GMT) (envelope-from noackjr@alumni.rice.edu) Received: from unknown (HELO optimator.noacks.org) (noacks@swbell.net@70.240.205.64 with login) by smtp812.mail.sc5.yahoo.com with SMTP; 3 May 2005 07:06:16 -0000 Received: from localhost (localhost [127.0.0.1]) by optimator.noacks.org (Postfix) with ESMTP id 82522614E; Tue, 3 May 2005 02:06:12 -0500 (CDT) Received: from optimator.noacks.org ([127.0.0.1]) by localhost (optimator.noacks.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 00566-12; Tue, 3 May 2005 02:06:07 -0500 (CDT) Received: from compgeek.noacks.org (compgeek [192.168.1.10]) by optimator.noacks.org (Postfix) with ESMTP id 024E060F7; Tue, 3 May 2005 02:06:06 -0500 (CDT) Received: from [127.0.0.1] (localhost [127.0.0.1]) by compgeek.noacks.org (8.13.3/8.13.3) with ESMTP id j4375vZ7001808; Tue, 3 May 2005 02:06:00 -0500 (CDT) (envelope-from noackjr@alumni.rice.edu) Message-ID: <427722D0.8050200@alumni.rice.edu> Date: Tue, 03 May 2005 02:05:52 -0500 From: Jonathan Noack User-Agent: Mozilla Thunderbird 1.0.2 (X11/20050428) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Petri Helenius References: <19879.1115061648@critter.freebsd.dk> <20050502214208.M87351@fledge.watson.org> <42770026.80901@he.iki.fi> In-Reply-To: <42770026.80901@he.iki.fi> X-Enigmail-Version: 0.91.0.0 OpenPGP: id=991D8195; url=http://www.noacks.org/cert/noackjr.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigAAE6D0C55B37F0B412973BCF" X-Virus-Scanned: amavisd-new at noacks.org cc: Steven Hartland cc: Poul-Henning Kamp cc: Robert Watson cc: freebsd-performance@FreeBSD.org cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: noackjr@alumni.rice.edu List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 07:06:18 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigAAE6D0C55B37F0B412973BCF Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 05/02/05 23:37, Petri Helenius wrote: > Robert Watson wrote: >> The next thing that would be quite nice to measure is the rate of I/O >> transactions per second we can get to the disk using the disk device >> directly, with a minimal transaction size. I have a vague >> recollection that you have to be careful in Linux because their >> character device nodes for disk devices are buffered, and you really >> want unbuffered I/O. > > I noticed that changing vfs.read_max from the default 8 to 16 has a > dramatic effect on sequential read performance. Increasing it further > did not have measurable effect. Wow. Setting vfs.read_max=16 resulted in a 50+% performance boost when sequentially reading from my 4-disk RAID5 (ida(4)) via the filesystem. Write performance was unchanged. Raw device read performance was also unchanged (~40 MB/s). With vfs.read_max>8, reading through the filesystem is *faster* than reading from the raw device! Note that these tests were run in multi-user on an idle 5.4-RC4 machine with 1 GB of RAM and quad P3 Xeons. The machine is currently using ULE. These tests were repeated 3 times with results in the +-0.1MB/s range. $ sysctl vfs.read_max vfs.read_max: 8 $ /usr/bin/time -h dd of=/dev/null if=/usr/testfile bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 185.571466 secs (35315774 bytes/sec) 3m5.58s real 0.16s user 1m0.89s sys $ sysctl vfs.read_max=16 vfs.read_max: 8 -> 16 $ /usr/bin/time -h dd of=/dev/null if=/usr/testfile bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 118.846665 secs (55143323 bytes/sec) 1m58.85s real 0.21s user 1m3.25s sys $ sysctl vfs.read_max=24 vfs.read_max: 16 -> 24 $ /usr/bin/time -h dd of=/dev/null if=/usr/testfile bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 107.360600 secs (61042878 bytes/sec) 1m47.36s real 0.14s user 1m3.27s sys $ sysctl vfs.read_max=32 vfs.read_max: 24 -> 32 $ /usr/bin/time -h dd of=/dev/null if=/usr/testfile bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 102.288822 secs (64069562 bytes/sec) 1m42.29s real 0.20s user 1m4.52s sys -- Jonathan Noack | noackjr@alumni.rice.edu | OpenPGP: 0x991D8195 --------------enigAAE6D0C55B37F0B412973BCF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (FreeBSD) iD8DBQFCdyLVUFz01pkdgZURAlDKAJwP9Q85zVvLODRJrTC25BYo+tdCeQCfYQmu OCQNhEc3+DOVFk90zEmoliw= =WR+k -----END PGP SIGNATURE----- --------------enigAAE6D0C55B37F0B412973BCF-- From owner-freebsd-performance@FreeBSD.ORG Tue May 3 07:44:05 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 49BF116A4CF for ; Tue, 3 May 2005 07:44:05 +0000 (GMT) Received: from mx2.mail.ru (mx2.mail.ru [194.67.23.122]) by mx1.FreeBSD.org (Postfix) with ESMTP id E675B43D58 for ; Tue, 3 May 2005 07:44:04 +0000 (GMT) (envelope-from samspeedu@mail.ru) Received: from [213.129.119.20] (port=1297 helo=192.168.168.7) by mx2.mail.ru with esmtp id 1DSs4Y-0006En-00 for freebsd-performance@freebsd.org; Tue, 03 May 2005 11:44:03 +0400 Date: Tue, 3 May 2005 11:43:55 +0400 From: Andrey Smagin X-Mailer: The Bat! (v1.62r) Organization: DiP X-Priority: 3 (Normal) Message-ID: <1534004626.20050503114355@mail.ru> To: freebsd-performance@freebsd.org In-Reply-To: <42770026.80901@he.iki.fi> References: <19879.1115061648@critter.freebsd.dk> <20050502214208.M87351@fledge.watson.org> <42770026.80901@he.iki.fi> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: SAMU List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 07:44:05 -0000 Hi ALL, I not sure that it only low disk performance, but. When i copying /usr/src tree from file server (AMD K6-2 225) to my Duron 1133(1GB RAM), speed is about 3-4MBytes/s. But played music on my PC is some gappy. Also removing /usr/ports tree have same much effect for music. I set large buffers in all avaliable players(mpg123, mpg321, mplayer, xine, xmms) on my system (FBSD6.0-current 1 may 05), but it don't have effect. I think that somthing lock IRQ from sound card(SBLive emu10k1 classic FBSD driver :) ) or call of SB buffer fill. How to diagnose what uninterruptable process(or in system function) can to lock CPU for so much time (1 sound buffer - 4096 bytes 4096/176400=~23ms) ? And, please, help me with tuning of sound performance. I think it depend from disk issues. -- Best regards, Andrey mailto:samspeedu@mail.ru From owner-freebsd-performance@FreeBSD.ORG Tue May 3 08:18:27 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7854C16A4CE for ; Tue, 3 May 2005 08:18:27 +0000 (GMT) Received: from web41201.mail.yahoo.com (web41201.mail.yahoo.com [66.218.93.34]) by mx1.FreeBSD.org (Postfix) with SMTP id 18FAB43D66 for ; Tue, 3 May 2005 08:18:27 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 72665 invoked by uid 60001); 3 May 2005 08:18:26 -0000 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; b=4Dtn+E+Q5wseKrePWOlYJmDysftG0sAuV6/5j6yMwv8Qn4nm04m3vbK9i3Z4c0Teg+jGJDpI1vgGRgEvmm4LgEt6zPeSXOfpk6mAVaoZIu2/2m4I09oZXFRQZ2ANoZslUpKA33mWiF6xaUwQUK7oZ0wRHI8R4yw+gtgI9C/XkxY= ; Message-ID: <20050503081826.72663.qmail@web41201.mail.yahoo.com> Received: from [83.129.182.168] by web41201.mail.yahoo.com via HTTP; Tue, 03 May 2005 01:18:26 PDT Date: Tue, 3 May 2005 01:18:26 -0700 (PDT) From: Arne "Wörner" To: SAMU , freebsd-performance@freebsd.org In-Reply-To: 6667 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 08:18:27 -0000 --- Andrey Smagin wrote: > driver :) ) or call of SB buffer fill. How to diagnose what > uninterruptable > process(or in system function) can to lock CPU for so much time > (1 sound buffer - 4096 bytes 4096/176400=~23ms) ? And, please, > help me with > tuning of sound performance. I think it depend from disk issues. > I can produce the same effect here with dd if=/dev/zero of=/dev/null bs=400k count=10000 (no disc involved). It gets very nasty with bs=1m -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Tue May 3 10:11:04 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BD24316A4D4 for ; Tue, 3 May 2005 10:11:04 +0000 (GMT) Received: from mx2.mail.ru (mx2.mail.ru [194.67.23.122]) by mx1.FreeBSD.org (Postfix) with ESMTP id 625EE43D91 for ; Tue, 3 May 2005 10:11:04 +0000 (GMT) (envelope-from samspeedu@mail.ru) Received: from [213.129.119.20] (port=1492 helo=192.168.168.7) by mx2.mail.ru with esmtp id 1DSuMi-0005kf-00 for freebsd-performance@freebsd.org; Tue, 03 May 2005 14:10:57 +0400 Date: Tue, 3 May 2005 14:10:50 +0400 From: Andrey Smagin X-Mailer: The Bat! (v1.62r) Organization: DiP X-Priority: 3 (Normal) Message-ID: <11110635136.20050503141050@mail.ru> To: freebsd-performance@freebsd.org In-Reply-To: <20050503081826.72663.qmail@web41201.mail.yahoo.com> References: <20050503081826.72663.qmail@web41201.mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: SAMU List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 10:11:04 -0000 AW> I can produce the same effect here with AW> dd if=/dev/zero of=/dev/null bs=400k count=10000 AW> (no disc involved). It gets very nasty with bs=1m I understand it, i want to know how to give more priority for sound subsystem instead another dev IO. -- Best regards, Andrey mailto:samspeedu@mail.ru From owner-freebsd-performance@FreeBSD.ORG Tue May 3 10:22:17 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E904C16A4CE for ; Tue, 3 May 2005 10:22:17 +0000 (GMT) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id 732ED43D70 for ; Tue, 3 May 2005 10:22:17 +0000 (GMT) (envelope-from kjelderg@gmail.com) Received: by rproxy.gmail.com with SMTP id j1so1030826rnf for ; Tue, 03 May 2005 03:22:17 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=VKFSK+ottPKAPomBHQKuQwyy9oOunWuwd51crsXi733T08qeEf+7EkqwN8F3FbT+1Zdn2uUarlZeUP0tqAliuR1rT5x+LYsbw6I6vT2LCG6e1H2ARO47Hxa3Pkq0OEoMOOj7ctllMi5tKbBsHKsaRP42eZjHRq88Ep7t4Fb3i9Q= Received: by 10.38.75.26 with SMTP id x26mr7366562rna; Tue, 03 May 2005 03:22:17 -0700 (PDT) Received: by 10.38.101.19 with HTTP; Tue, 3 May 2005 03:22:17 -0700 (PDT) Message-ID: Date: Tue, 3 May 2005 19:22:17 +0900 From: Eric Kjeldergaard To: Steven Hartland In-Reply-To: <003901c54f36$0c64ad40$b3db87d4@multiplay.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <17813.1115042014@critter.freebsd.dk> <003901c54f36$0c64ad40$b3db87d4@multiplay.co.uk> cc: Robert Watson cc: Poul-Henning Kamp cc: Eric Anderson cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Eric Kjeldergaard List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 10:22:18 -0000 > All partitions / file system creation was done using the OS default > tool i.e. FreeBSD: sysinstall, Suse: yast Perhaps the defaults are a bit dangerous. If you have the time, could you post results with freebsd mounting asynchronously (add async as an option in the fstab) and linux mounting synchronously (again, sync in fstab as an option) as I believe these to be the opposite of the defaults. I, for one, would appreciate those results. --=20 If I write a signature, my emails will appear more personalised. From owner-freebsd-performance@FreeBSD.ORG Tue May 3 11:45:34 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 018AB16A4CE; Tue, 3 May 2005 11:45:34 +0000 (GMT) Received: from ms1.as.pvp.se (dns.pvp.se [213.64.187.226]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4BACD43D1D; Tue, 3 May 2005 11:45:33 +0000 (GMT) (envelope-from kama@pvp.se) Received: by ms1.as.pvp.se (Postfix, from userid 1001) id 57C48A7; Tue, 3 May 2005 13:45:30 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by ms1.as.pvp.se (Postfix) with ESMTP id 537E7A6; Tue, 3 May 2005 13:45:30 +0200 (CEST) Date: Tue, 3 May 2005 13:45:30 +0200 (CEST) From: kama X-X-Sender: kama@ns1.as.pvp.se To: Poul-Henning Kamp In-Reply-To: <17442.1115039706@critter.freebsd.dk> Message-ID: <20050503134451.P22614@ns1.as.pvp.se> References: <17442.1115039706@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Eric Anderson cc: =?ISO-8859-15?Q?=22Arne_=5C=22W=F6rner=5C=22=22?= Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 11:45:34 -0000 On Mon, 2 May 2005, Poul-Henning Kamp wrote: > In message <427626DC.5030702@centtech.com>, Eric Anderson writes: > > >Don't mean to be terse here, but I'm talking about the same test done an > >two different RAID5 configurations, with different disks, and not just > >me - other users in this very thread see the same issue.. > > Uhm, if you are using RAID5 and your requests are not aligned and > sized after the RAID5 you should *expect* read performance to be poor. > > If you your request ends up accessing two different blocks even just > once per stripe, this totally kills performance. My system is setup with RAID 1+0. So its not a RAID5 issue... From owner-freebsd-performance@FreeBSD.ORG Tue May 3 14:53:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BB78F16A4CE; Tue, 3 May 2005 14:53:10 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id AF5DA43D86; Tue, 3 May 2005 14:53:09 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001378031.msg; Tue, 03 May 2005 15:48:21 +0100 Message-ID: <01ad01c54fef$a56e6260$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Petri Helenius" , "Robert Watson" References: <19879.1115061648@critter.freebsd.dk><20050502214208.M87351@fledge.watson.org> <42770026.80901@he.iki.fi> Date: Tue, 3 May 2005 15:51:50 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Tue, 03 May 2005 15:48:21 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDAV-Processed: multiplay.co.uk, Tue, 03 May 2005 15:48:22 +0100 cc: Poul-Henning Kamp cc: Eric Anderson cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 May 2005 14:53:10 -0000 Summary of results: RAID0: Changing vfs.read_max 8 -> 16 and MAXPHYS 128k -> 1M increased read performance significantly from 129Mb/s to 199MB/s Max raw device speed here was 234Mb/s FS -> Raw device: 35Mb/s 14.9% performance loss RAID5: Changing vfs.read_max 8 -> 16 produced a small increase 129Mb/s to 135Mb/s. Increasing MAXPHYS 128k -> 1M prevented vfs.read_max from having any effect Max raw device speed here was 200Mb/s FS -> Raw device: 65Mb/s 32.5% performance loss Note: This batch of tests where done on uni processor kernel to keep variation down to a minimum, so are not directly comparable with my previous tests. All tests where performed with 16k RAID stripe across all 5 disks and a default newfs. Increasing or decreasing the block size for the fs was tried but only had negative effects. Results: **RAID0** sysctl vfs.read_max=8 dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 48.459904 secs (135237577 bytes/sec) sysctl vfs.read_max=16 dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 35.338873 secs (185450169 bytes/sec) sysctl vfs.read_max=24 dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 35.692228 secs (183614203 bytes/sec) sysctl vfs.read_max=32 dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 35.694294 secs (183603576 bytes/sec) sysctl vfs.read_max=8 + MAXPHYS = 1M dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 40.546771 secs (161630626 bytes/sec) sysctl vfs.read_max=16 + MAXPHYS = 1M dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 31.698017 secs (206751103 bytes/sec) sysctl vfs.read_max=24 + MAXPHYS = 1M dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 31.409248 secs (208651924 bytes/sec) dd if=/mnt/testfile of=/dev/null bs=32k count=200000 200000+0 records in 200000+0 records out 6553600000 bytes transferred in 31.981614 secs (204917739 bytes/sec dd if=/mnt/testfile of=/dev/null bs=128k count=50000 50000+0 records in 50000+0 records out 6553600000 bytes transferred in 31.277051 secs (209533821 bytes/sec) dd if=/mnt/testfile of=/dev/null bs=256k count=25000 25000+0 records in 25000+0 records out 6553600000 bytes transferred in 33.271369 secs (196974161 bytes/sec) dd if=/mnt/testfile of=/dev/null bs=512k count=12500 12500+0 records in 12500+0 records out 6553600000 bytes transferred in 35.499299 secs (184612096 bytes/sec) dd if=/mnt/testfile of=/dev/null bs=1024k count=6250 6250+0 records in 6250+0 records out 6553600000 bytes transferred in 36.861447 secs (177790090 bytes/sec) sysctl vfs.read_max=24 + MAXPHYS = 1M ( raw device ) dd if=/dev/da0 of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 27.818829 secs (235581447 bytes/sec) dd if=/dev/da0 of=/dev/null bs=1024k count=6250 6250+0 records in 6250+0 records out 6553600000 bytes transferred in 26.610258 secs (246280963 bytes/sec) **RAID5** sysctl vfs.read_max=8 ( raw device ) dd if=/dev/da0 of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 31.110141 secs (210657997 bytes/sec) dd if=/dev/da0 of=/dev/null bs=1024k count=6520 6520+0 records in 6520+0 records out 6836715520 bytes transferred in 31.147035 secs (219498116 bytes/sec) sysctl vfs.read_max=8 dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 48.380458 secs (135459652 bytes/sec) sysctl vfs.read_max=16 dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 46.318109 secs (141491096 bytes/sec) sysctl vfs.read_max=24 dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 46.305371 secs (141530018 bytes/sec) sysctl vfs.read_max=8 + MAXPHYS = 1M dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 49.093645 secs (133491819 bytes/sec) sysctl vfs.read_max=16 + MAXPHYS = 1M dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 48.347905 secs (135550857 bytes/sec) sysctl vfs.read_max=24 + MAXPHYS = 1M dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 48.702546 secs (134563807 bytes/sec) sysctl vfs.read_max=24 + MAXPHYS = 1M ( raw device ) dd if=/dev/da0 of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 38.822286 secs (168810254 bytes/sec) dd if=/dev/da0 of=/dev/null bs=1024k count=6250 6250+0 records in 6250+0 records out 6553600000 bytes transferred in 38.727828 secs (169221987 bytes/sec) ----- Original Message ----- From: "Petri Helenius" > > I noticed that changing vfs.read_max from the default 8 to 16 has a > dramatic effect on sequential read performance. Increasing it further > did not have measurable effect. ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Wed May 4 13:06:58 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5256E16A4D1 for ; Wed, 4 May 2005 13:06:58 +0000 (GMT) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4604F43D64 for ; Wed, 4 May 2005 13:06:57 +0000 (GMT) (envelope-from mike@sentex.net) Received: from pumice5.sentex.ca (pumice5.sentex.ca [64.7.153.20]) by smarthost1.sentex.ca (8.13.3/8.13.3) with ESMTP id j44BvP8j067432 for ; Wed, 4 May 2005 07:57:25 -0400 (EDT) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by pumice5.sentex.ca (8.13.3/8.13.1) with ESMTP id j44BvTVa009478; Wed, 4 May 2005 07:57:29 -0400 (EDT) (envelope-from mike@sentex.net) Received: from simian.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.3/8.12.11) with ESMTP id j44BvFlT001611; Wed, 4 May 2005 07:57:16 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <6.2.1.2.0.20050502222250.02da34f8@64.7.153.2> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Wed, 04 May 2005 07:57:15 -0400 To: "Poul-Henning Kamp" From: Mike Tancsa In-Reply-To: <20925.1115069492@critter.freebsd.dk> References: <20925.1115069492@critter.freebsd.dk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: by amavisd-new X-Scanned-By: MIMEDefang 2.51 on 64.7.153.20 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x (some tweaking and more results) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2005 13:06:58 -0000 At 05:31 PM 02/05/2005, Poul-Henning Kamp wrote: >In message <6.2.1.2.0.20050502163426.02c4e678@64.7.153.2>, Mike Tancsa writes: > > >Using an amrc controller in RAID5, it doesnt make a difference really on > >the dd stuff - read or write-- perhaps 2-5MB difference on the faster > >side. Raw reads and writes whether I use da0 or da0s1d, the speed is > >pretty well the same as before... HOWEVER, one measurement is really odd on > >bonnie-- the random seeks .... and postmark shows a *big* difference as > well. > >This is very likely the alignment thing. OK, some further tests, trying to control for this. I am not sure what values to actually fiddle with via bsdlable as this is the entire disk so I will just vary mounting da0 vs da0s1d. I also cvsup'd to current as of yesterday (FreeBSD nfs.sentex.ca 6.0-CURRENT FreeBSD 6.0-CURRENT #0: Mon May 2 15:03:53 EDT 2005). I will also try with Scott's changes to the driver which went in after. Also, the newfs params does seem to make a very big difference and vfs.read_max as well. [nfs]# fdisk da0 ******* Working on device /dev/da0 ******* parameters extracted from in-core disklabel are: cylinders=91182 heads=255 sectors/track=63 (16065 blks/cyl) Figures below won't work with BIOS for partitions not in cyl 1 parameters to be used for BIOS calculations are: cylinders=91182 heads=255 sectors/track=63 (16065 blks/cyl) Media sector size is 512 Warning: BIOS sector numbering starts with sector 1 Information from DOS bootblock is: The data for partition 1 is: sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD) start 63, size 1464838767 (715253 Meg), flag 80 (active) beg: cyl 0/ head 1/ sector 1; end: cyl 1023/ head 254/ sector 63 The data for partition 2 is: The data for partition 3 is: The data for partition 4 is: [nfs]# bsdlabel da0s1 # /dev/da0s1: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 1464838767 0 unused 0 0 # "raw" part, don't edit d: 1464838767 0 4.2BSD 2048 16384 28552 [nfs]# newfs -b 32768 -f 4096 /dev/da0 [nfs]# bsdlabel da0 bsdlabel: /dev/da0: no valid label found [nfs]# even though I can mount it just fine [nfs]# mount /dev/da0 /mnt [nfs]# df Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/ad0s1a 1012974 356028 575910 38% / devfs 1 1 0 100% /dev /dev/ad0s1e 1012974 580870 351068 62% /tmp /dev/ad0s1d 15231278 5848016 8164760 42% /usr /dev/ad0s1f 19564270 192176 17806954 1% /var /dev/da0 720910352 8 663237516 0% /mnt [nfs]# -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU da0s1 4000 61175 53.3 61832 16.3 31829 8.7 63008 60.4 79551 11.2 224.2 0.7 da0s1 v 4000 61278 54.3 61938 16.4 31828 8.8 73566 71.7 81778 12.6 216.5 0.7 da0 4000 62625 54.5 62154 15.7 31907 8.3 63297 60.8 80139 11.0 226.2 0.7 da0 v 4000 62315 54.3 61759 15.4 32094 8.3 74209 72.3 84470 12.6 223.5 0.7 da0 N 4000 75239 62.7 75771 15.5 33596 8.4 81909 78.6 98453 13.3 214.8 0.8 da0 vN 4000 72786 61.7 74939 15.3 33672 8.4 82944 79.8 108106 14.9 214.7 0.8 v = sysctl -w vfs.read_max=16 N = newfs -b 32768 -f 4096 /dev/da0 [nfs]# postmark PostMark v1.5 : 3/27/01 pm>set location /mnt pm>set size 300 100000 pm>set transactions 400000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 1418 seconds total 1418 seconds of transactions (282 per second) Files: 200107 created (141 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (140 per second) 199905 read (140 per second) 199384 appended (140 per second) 200107 deleted (141 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (140 per second) Data: 12715.55 megabytes read (8.97 megabytes per second) 12728.92 megabytes written (8.98 megabytes per second) pm>quit [nfs]# create a junk file [nfs]# time dd if=/dev/urandom of=/mnt/big-4g-rand bs=1024k count=4000 4000+0 records in 4000+0 records out 4194304000 bytes transferred in 124.694915 secs (33636528 bytes/sec) 0.022u 116.020s 2:04.70 93.0% 21+2910k 124+32000io 0pf+0w wait 2 min for disks to totally flush [nfs]# time dd if=/dev/zero of=/mnt/big-4g-zero bs=1024k count=4000 4000+0 records in 4000+0 records out 4194304000 bytes transferred in 66.463634 secs (63106751 bytes/sec) 0.000u 11.265s 1:06.50 16.9% 21+2926k 126+32000io 0pf+0w sysctl -w vfs.read_max=16 (clear out cache by reading via md5 4G of other files) and then wait a bit [nfs]# time dd if=/mnt/big-4g-rand of=/dev/null bs=1024k count=4000 4000+0 records in 4000+0 records out 4194304000 bytes transferred in 59.178470 secs (70875506 bytes/sec) 0.000u 7.677s 0:59.18 12.9% 20+2784k 32128+0io 0pf+0w [nfs]# postmark PostMark v1.5 : 3/27/01 pm>set locatime /mnt Eh? pm>set location /mnt pm>set size 300 100000 pm>set transactions 400000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 1369 seconds total 1369 seconds of transactions (292 per second) Files: 200107 created (146 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (145 per second) 199905 read (146 per second) 199384 appended (145 per second) 200107 deleted (146 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (145 per second) Data: 12715.55 megabytes read (9.29 megabytes per second) 12728.92 megabytes written (9.30 megabytes per second) pm> With vfs.read_max=8 [nfs]# postmark PostMark v1.5 : 3/27/01 pm>set location /mnt pm>set size 300 100000 pm>set transactions 400000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 1013 seconds total 1013 seconds of transactions (394 per second) Files: 200107 created (197 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (197 per second) 199905 read (197 per second) 199384 appended (196 per second) 200107 deleted (197 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (196 per second) Data: 12715.55 megabytes read (12.55 megabytes per second) 12728.92 megabytes written (12.57 megabytes per second) pm> pm>quit [nfs]# --------------------------------------------------------------------- newfs -b 32768 -f 4096 /dev/da0 [nfs]# postmark PostMark v1.5 : 3/27/01 pm>set location /mnt pm>set transactions 400000 pm>set size 300 100000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 542 seconds total 541 seconds of transactions (739 per second) Files: 200107 created (369 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (368 per second) 199905 read (369 per second) 199384 appended (368 per second) 200107 deleted (369 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (368 per second) Data: 12715.55 megabytes read (23.46 megabytes per second) 12728.92 megabytes written (23.49 megabytes per second) pm> [nfs]# sysctl -w vfs.read_max=16 vfs.read_max: 8 -> 16 pm>set location /mnt pm>set transactions 400000 pm>set size 300 100000 pm>run Creating files...Done Performing transactions..........Done Deleting files...Done Time: 542 seconds total 541 seconds of transactions (739 per second) Files: 200107 created (369 per second) Creation alone: 500 files (500 per second) Mixed with transactions: 199607 files (368 per second) 199905 read (369 per second) 199384 appended (368 per second) 200107 deleted (369 per second) Deletion alone: 889 files (889 per second) Mixed with transactions: 199218 files (368 per second) Data: 12715.55 megabytes read (23.46 megabytes per second) 12728.92 megabytes written (23.49 megabytes per second) pm> [nfs]# time dd if=/mnt/big-2g-rand count=2000 bs=1024k of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 19.298799 secs (108667489 bytes/sec) 0.000u 3.603s 0:19.32 18.6% 21+2965k 16019+0io 0pf+0w [nfs]# time dd if=/mnt/big-2g-rand2 count=2000 bs=1024k of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 20.188331 secs (103879414 bytes/sec) 0.015u 3.515s 0:20.24 17.3% 20+2865k 16020+0io 3pf+0w [nfs]# sysctl -w vfs.read_max=8 vfs.read_max: 16 -> 8 [nfs]# time dd if=/mnt/big-2g-rand count=2000 bs=1024k of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 21.421867 secs (97897722 bytes/sec) 0.007u 3.482s 0:21.43 16.2% 21+2873k 16018+0io 0pf+0w [nfs]# time dd if=/mnt/big-2g-rand2 count=2000 bs=1024k of=/dev/null 2000+0 records in 2000+0 records out 2097152000 bytes transferred in 23.306603 secs (89981024 bytes/sec) 0.000u 3.590s 0:23.37 15.3% 19+2687k 16020+0io 3pf+0w [nfs]# From owner-freebsd-performance@FreeBSD.ORG Wed May 4 14:04:53 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 85F4716A4CE for ; Wed, 4 May 2005 14:04:53 +0000 (GMT) Received: from multiplay.co.uk (www1.multiplay.co.uk [212.42.16.7]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5004043D4C for ; Wed, 4 May 2005 14:04:52 +0000 (GMT) (envelope-from killing@multiplay.co.uk) Received: from vader ([212.135.219.179]) by multiplay.co.uk (multiplay.co.uk [212.42.16.7]) (MDaemon.PRO.v8.0.1.R) with ESMTP id md50001380666.msg for ; Wed, 04 May 2005 15:00:46 +0100 Message-ID: <00db01c550b2$2a598dc0$b3db87d4@multiplay.co.uk> From: "Steven Hartland" To: "Poul-Henning Kamp" , "Mike Tancsa" References: <20925.1115069492@critter.freebsd.dk> <6.2.1.2.0.20050502222250.02da34f8@64.7.153.2> Date: Wed, 4 May 2005 15:04:15 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2527 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2527 X-Spam-Processed: multiplay.co.uk, Wed, 04 May 2005 15:00:46 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 212.135.219.179 X-Return-Path: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org X-MDAV-Processed: multiplay.co.uk, Wed, 04 May 2005 15:00:48 +0100 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x (some tweaking and more results) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 May 2005 14:04:53 -0000 Did you also try the sys/param.h change that helped here. Also when testing on FS I found bs=1024k to degrade performance try with 64k. Is this a raid volume? If so on my setup anything other that a 16k stripe and performance went out the window. For the 'time' its easier to understand if u use: /usr/bin/time -h Last but not least. I found some very strange behaviour late last night. If I created the raid set but didnt power down after doing so the results where a lot lower than when I did. Dont ask me why I have no idea, only noticed as I retested a result after I installed a new fan in the machine and was totally a los My initial starting results where: Write: 140Mb/s Read: 49Mb/s My current values are: Write: 140Mb/s Read: 195Mb/s Changes made: 1. 16k RAID5 stripe not 64k ( default ) 2. vfs.read_max=16 3. MAXPHYS = 256 ( was 128 ) 4. newfs /dev/da0 ( was a basic install with multi partitions ) I'm currently seeing some VERY strange behaviour which could be the RAID controller negotiating different PCI-X speeds. Anyone know how to check the state of a PCI-X card? Power up, login: dd if=/dev/da0 of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 42.997295 secs (152418891 bytes/sec) mount /dev/da0 /mnt dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 48.757091 secs (134413270 bytes/sec) shutdown -p now Power up, login: dd if=/dev/da0 of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 28.365671 secs (231039838 bytes/sec) mount /dev/da0 /mnt dd if=/mnt/testfile of=/dev/null bs=64k count=100000 100000+0 records in 100000+0 records out 6553600000 bytes transferred in 32.012170 secs (204722143 bytes/sec) Steve ----- Original Message ----- From: "Mike Tancsa" > OK, some further tests, trying to control for this. I am not sure what > values to actually fiddle with via bsdlable as this is the entire disk so I > will just vary mounting da0 vs da0s1d. I also cvsup'd to current as of > yesterday (FreeBSD nfs.sentex.ca 6.0-CURRENT FreeBSD 6.0-CURRENT #0: Mon > May 2 15:03:53 EDT 2005). I will also try with Scott's changes to the > driver which went in after. > > Also, the newfs params does seem to make a very big difference and > vfs.read_max as well. .... ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone (023) 8024 3137 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Fri May 6 16:04:25 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5E1A116A4D4 for ; Fri, 6 May 2005 16:04:25 +0000 (GMT) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id A766443D88 for ; Fri, 6 May 2005 16:04:24 +0000 (GMT) (envelope-from mike@sentex.net) Received: from spamscanner3.sentex.ca (pumice3.sentex.ca [64.7.153.26]) by smarthost1.sentex.ca (8.13.3/8.13.3) with ESMTP id j46G4Ocu047299 for ; Fri, 6 May 2005 12:04:24 -0400 (EDT) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) j46G4Ndn019785; Fri, 6 May 2005 12:04:23 -0400 (EDT) (envelope-from mike@sentex.net) Received: from simian.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.3/8.12.11) with ESMTP id j46G4FjN010916; Fri, 6 May 2005 12:04:16 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <6.2.1.2.0.20050506115815.07dab518@64.7.153.2> X-Mailer: QUALCOMM Windows Eudora Version 6.2.1.2 Date: Fri, 06 May 2005 12:03:54 -0400 To: "Steven Hartland" , "Poul-Henning Kamp" From: Mike Tancsa In-Reply-To: <00db01c550b2$2a598dc0$b3db87d4@multiplay.co.uk> References: <20925.1115069492@critter.freebsd.dk> <6.2.1.2.0.20050502222250.02da34f8@64.7.153.2> <00db01c550b2$2a598dc0$b3db87d4@multiplay.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: by amavisd-new X-Scanned-By: MIMEDefang 2.49 on 64.7.153.26 cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x (some tweaking and more results) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 May 2005 16:04:25 -0000 At 10:04 AM 04/05/2005, Steven Hartland wrote: >Did you also try the sys/param.h change that helped here. I have not yet but will soon. The tweaking certainly does make a difference in various numbers. I ran some extensive iozone tests overnight too see how those numbers are effected by these tweaks. But in terms of raw reads, here is the range. This is the same hardware, just varying how the disk is mounted, the vfs readmax and the newfs params 4194304000 bytes transferred in 65.338981 secs (64192982 bytes/sec) 4194304000 bytes transferred in 53.246965 secs (78770762 bytes/sec) 4194304000 bytes transferred in 62.046088 secs (67599814 bytes/sec) 4194304000 bytes transferred in 55.313732 secs (75827536 bytes/sec) 4194304000 bytes transferred in 59.167997 secs (70888051 bytes/sec) 4194304000 bytes transferred in 56.293913 secs (74507238 bytes/sec) 4194304000 bytes transferred in 53.891288 secs (77828980 bytes/sec) 4194304000 bytes transferred in 58.828609 secs (71297011 bytes/sec) 4194304000 bytes transferred in 54.110452 secs (77513749 bytes/sec) 4194304000 bytes transferred in 54.480602 secs (76987108 bytes/sec) 4194304000 bytes transferred in 54.604255 secs (76812769 bytes/sec) 4194304000 bytes transferred in 53.150221 secs (78914141 bytes/sec) 4194304000 bytes transferred in 63.662145 secs (65883799 bytes/sec) 4194304000 bytes transferred in 54.131878 secs (77483068 bytes/sec) 4194304000 bytes transferred in 59.093488 secs (70977432 bytes/sec) 4194304000 bytes transferred in 51.723489 secs (81090895 bytes/sec) 4194304000 bytes transferred in 48.060447 secs (87271431 bytes/sec) 4194304000 bytes transferred in 58.114882 secs (72172632 bytes/sec) 4194304000 bytes transferred in 36.197407 secs (115873051 bytes/sec) 4194304000 bytes transferred in 38.453472 secs (109074780 bytes/sec) 4194304000 bytes transferred in 39.662853 secs (105748923 bytes/sec) 4194304000 bytes transferred in 35.170596 secs (119255983 bytes/sec) 4194304000 bytes transferred in 35.173053 secs (119247652 bytes/sec) 4194304000 bytes transferred in 35.241742 secs (119015230 bytes/sec) I should get the iozone stuff summarized over the weekend. ---Mike >Also when testing on FS I found bs=1024k to degrade performance >try with 64k. >Is this a raid volume? If so on my setup anything other that a 16k stripe >and performance went out the window. > >For the 'time' its easier to understand if u use: >/usr/bin/time -h > >Last but not least. I found some very strange behaviour late last night. >If I created the raid set but didnt power down after doing so the >results where a lot lower than when I did. Dont ask me why I have >no idea, only noticed as I retested a result after I installed a new >fan in the machine and was totally a los > >My initial starting results where: >Write: 140Mb/s >Read: 49Mb/s > >My current values are: >Write: 140Mb/s >Read: 195Mb/s > >Changes made: >1. 16k RAID5 stripe not 64k ( default ) >2. vfs.read_max=16 >3. MAXPHYS = 256 ( was 128 ) >4. newfs /dev/da0 ( was a basic install with multi partitions ) > >I'm currently seeing some VERY strange behaviour which could >be the RAID controller negotiating different PCI-X speeds. > >Anyone know how to check the state of a PCI-X card? > >Power up, login: >dd if=/dev/da0 of=/dev/null bs=64k count=100000 >100000+0 records in >100000+0 records out >6553600000 bytes transferred in 42.997295 secs (152418891 bytes/sec) >mount /dev/da0 /mnt >dd if=/mnt/testfile of=/dev/null bs=64k count=100000 >100000+0 records in >100000+0 records out >6553600000 bytes transferred in 48.757091 secs (134413270 bytes/sec) >shutdown -p now > >Power up, login: >dd if=/dev/da0 of=/dev/null bs=64k count=100000 >100000+0 records in >100000+0 records out >6553600000 bytes transferred in 28.365671 secs (231039838 bytes/sec) >mount /dev/da0 /mnt >dd if=/mnt/testfile of=/dev/null bs=64k count=100000 >100000+0 records in >100000+0 records out >6553600000 bytes transferred in 32.012170 secs (204722143 bytes/sec) > > Steve >----- Original Message ----- From: "Mike Tancsa" >>OK, some further tests, trying to control for this. I am not sure what >>values to actually fiddle with via bsdlable as this is the entire disk so >>I will just vary mounting da0 vs da0s1d. I also cvsup'd to current as >>of yesterday (FreeBSD nfs.sentex.ca 6.0-CURRENT FreeBSD 6.0-CURRENT #0: >>Mon May 2 15:03:53 EDT 2005). I will also try with Scott's changes to >>the driver which went in after. >>Also, the newfs params does seem to make a very big difference and >>vfs.read_max as well. >.... > > >================================================ >This e.mail is private and confidential between Multiplay (UK) Ltd. and >the person or entity to whom it is addressed. In the event of >misdirection, the recipient is prohibited from using, copying, printing or >otherwise disseminating it or any information contained in it. >In the event of misdirection, illegible or incomplete transmission please >telephone (023) 8024 3137 >or return the E.mail to postmaster@multiplay.co.uk. > >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"