From owner-freebsd-performance@FreeBSD.ORG Tue May 5 05:57:32 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 197C1106566B for ; Tue, 5 May 2009 05:57:32 +0000 (UTC) (envelope-from markwkm@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.31]) by mx1.freebsd.org (Postfix) with ESMTP id CD40E8FC1A for ; Tue, 5 May 2009 05:57:31 +0000 (UTC) (envelope-from markwkm@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so2678651ywe.13 for ; Mon, 04 May 2009 22:57:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=byt733mtlmMjDy0Q/2V4Tv46FOWqQOq4vcld/2zrBvg=; b=PC+fF65as9qZD9MnKKlRvkHcxFjrk970TcAVhRTtteErIBDNa9Y8jasV2k9fSl6mv+ VhDcazb650flxs4p996EZwNUFiUmmoleEXAq1k9qe70gx5NeSuaTea3bjYGsiWtSqYMc ZFOPonGglU3PJAiBHRATsiksAhC4yvX/sOMSk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=INPYJjplcjO8wNYU9sgrP7GTgZLcQRwDz3Yva6XAmwm7KLzvZHGfkqmRdi+S7u5WYx TNYnsjVMMC06gG7NZFDhz18fnh1sHa9wDY/aqFP5U3bKXyMfiwx5FFYO6V7D4HtQsol/ NQ6GBmaJDOgNCiH+eR344r3VnK+ca0LJ9vS9I= MIME-Version: 1.0 Received: by 10.100.240.9 with SMTP id n9mr14903358anh.135.1241501447799; Mon, 04 May 2009 22:30:47 -0700 (PDT) Date: Mon, 4 May 2009 22:30:47 -0700 Message-ID: <70c01d1d0905042230v3357622cgf4c8e52a2a4ead96@mail.gmail.com> From: Mark Wong To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Selena Deckelmann , Gabrielle Roth Subject: filesystem performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2009 05:57:32 -0000 Hi everyone, We (PostgreSQL community) have a HP DL380 G5 that we were using to do some very basic filesystem characterizations as part of a database performance tuning project, so we wanted to give FreeBSD a try out of the box. For this set of data we used 7.1. We're (us few that are running the tests) are fairly unfamiliar with the community here, so I'll be as brief as I can. We're basically wondering if the data we're getting out of the box is expected, and any tuning guidelines including what changes we should expect to see in the performance. So that said, I'll start with some of the charted results: Scaling from 1 device to 4 devices using RAID 0: http://207.173.203.223/~markwkm/community10/fio/freebsd-7.1/raid0-ufs.png Comparing RAID configurations of similar capacity: http://207.173.203.223/~markwkm/community10/fio/freebsd-7.1/capacity-ufs.png Comparing RAID configuration using same number of drives (4): http://207.173.203.223/~markwkm/community10/fio/freebsd-7.1/4-disk-ufs.png All of the raw data fio output, iostat, and vmstat including charts of some of the iostat and vmstat data are buried in here: http://207.173.203.223/~markwkm/community10/fio/freebsd-7.1/ We used fio v1.23 to run the tests and picked parameters based on what PostgreSQL is capable of. The profiles used are here: http://git.postgresql.org/gitweb?p=performance-tuning.git;a=tree;f=contrib/freebsd/fio;h=5ae97420fba010a3685c12c24cd69266d83b4daf;hb=HEAD Hardware details are here (note we only used 4 of the internal drives for this test, none in the MSA 70, and you may notice we have a lit of Linux data on this page too): http://wiki.postgresql.org/wiki/HP_ProLiant_DL380_G5_Tuning_Guide#Hardware_Details I know I've been brief and at the same time dumped a lot of raw data. Any pointers to tuning guides that we have obviously not seen would be a appreciated. We hope this data is of interest and is helpful. Regards, Mark From owner-freebsd-performance@FreeBSD.ORG Tue May 5 15:39:24 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5798E1065674 for ; Tue, 5 May 2009 15:39:24 +0000 (UTC) (envelope-from ssanders@opnet.com) Received: from smtp-hq2.opnet.com (smtp-hq2.opnet.com [192.104.65.247]) by mx1.freebsd.org (Postfix) with ESMTP id 2EEA18FC15 for ; Tue, 5 May 2009 15:39:23 +0000 (UTC) (envelope-from ssanders@opnet.com) Received: from [172.16.12.251] (wtn12251.opnet.com [172.16.12.251]) by smtp.opnet.com (Postfix) with ESMTPSA id BF79E21100A5 for ; Tue, 5 May 2009 11:20:21 -0400 (EDT) Message-ID: <4A005935.40206@opnet.com> Date: Tue, 05 May 2009 11:20:21 -0400 From: Stephen Sanders Organization: OpNet User-Agent: Thunderbird 2.0.0.21 (Windows/20090302) MIME-Version: 1.0 To: freebsd-performance@freebsd.org X-Enigmail-Version: 0.95.7 Content-Type: multipart/mixed; boundary="------------080206040101010403070406" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: VM sysctl tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Stephen Sanders List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2009 15:39:24 -0000 This is a multi-part message in MIME format. --------------080206040101010403070406 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Does any know if there is any advantage to tuning sysctl's for the VM when one has an insane amount of memory in their machine? We've a system with 16GB of RAM that we are attempting to optimize the system's memory copy and disk write performance. Thanks. --------------080206040101010403070406-- From owner-freebsd-performance@FreeBSD.ORG Tue May 5 18:26:46 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 656DA1065688 for ; Tue, 5 May 2009 18:26:46 +0000 (UTC) (envelope-from leccine@gmail.com) Received: from mail-ew0-f159.google.com (mail-ew0-f159.google.com [209.85.219.159]) by mx1.freebsd.org (Postfix) with ESMTP id DD6098FC17 for ; Tue, 5 May 2009 18:26:45 +0000 (UTC) (envelope-from leccine@gmail.com) Received: by ewy3 with SMTP id 3so62487ewy.43 for ; Tue, 05 May 2009 11:26:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=rQl4o6ZDc/oWtPjDv491UW0HhdSlbgACcAD4hTLC7Ic=; b=Z1w2HdBIaNitlNh3aZf+KBL8eM6wrSgWicc4NABU4IF7XvlO1HOZenG1knt6V6u03E NfF2vjDrkYPQQhDbV1MNAlZ13dUtL+r2HjiO5/pK69sb27USgfFx/2YfUjwFAJB4YTv5 2TFkMd9b1juPgnekgGESkwYJ9aE7tWXQLaorE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=sAcYuaTS4cK8lzXAdGxUDRQImj0m2SCO84nbhOdRcB5hHIZzkD61n0KzdMAlAwOlTx P0kdSGkQCeLbgkO6y2vay15BBDDFCKWwCxbsgCjwE68VkXTlW4QQ5rzfRJYySrWYLAKY 6GT3iXG7CdhdvLZnffKA6T7qKigoiQSl4ZwP0= MIME-Version: 1.0 Received: by 10.210.58.17 with SMTP id g17mr461195eba.20.1241546663366; Tue, 05 May 2009 11:04:23 -0700 (PDT) In-Reply-To: <4A005935.40206@opnet.com> References: <4A005935.40206@opnet.com> Date: Tue, 5 May 2009 19:04:23 +0100 Message-ID: From: =?ISO-8859-1?B?SXN0duFu?= To: Stephen Sanders Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: VM sysctl tuning X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2009 18:26:46 -0000 I guess you can find some useful information here: http://wiki.freebsd.org/ZFSTuningGuide Please note the difference between 7.2 and all the previous releases. "FreeBSD 7.2+ has improved kernel memory allocation strategy and no tuning may be necessary on systems with more than 2 GB of RAM. " Also wort play a bit with kern.maxvnodes as the document says. Not to forget: http://www.freebsd.org/doc/en/books/arch-handbook/vm-tuning.html Regards, Istvan On Tue, May 5, 2009 at 4:20 PM, Stephen Sanders wrote: > Does any know if there is any advantage to tuning sysctl's for the VM > when one has an insane amount of memory in their machine? > > We've a system with 16GB of RAM that we are attempting to optimize the > system's memory copy and disk write performance. > > Thanks. > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > -- the sun shines for all From owner-freebsd-performance@FreeBSD.ORG Wed May 6 09:50:03 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35FEF1065673 for ; Wed, 6 May 2009 09:50:03 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx0.awanti.com (mx0.awanti.com [91.190.112.18]) by mx1.freebsd.org (Postfix) with ESMTP id E22D68FC19 for ; Wed, 6 May 2009 09:50:02 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from pstation (unknown [10.28.4.14]) by mx0.awanti.com (Postfix) with ESMTP id 5B0524C0B1; Wed, 6 May 2009 13:24:58 +0400 (MSD) Date: Wed, 6 May 2009 13:25:50 +0400 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <5871156390.20090506132550@mail.ru> To: Mark Wong In-Reply-To: <70c01d1d0905042230v3357622cgf4c8e52a2a4ead96@mail.gmail.com> References: <70c01d1d0905042230v3357622cgf4c8e52a2a4ead96@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org Subject: Re: filesystem performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 09:50:04 -0000 Hello Mark, May i ask a question while more expierenced people is waking up? I don't fully understand the target. For what filesystem should be optimized? I expect a patterns of recorded IO calls when pgsql perform typical operations with statistics and in-depth analysis. Are you sure there is à strong relation between fio benchmark result and PostgreSQL performance? Tuesday, May 05, 2009, 9:30:47 AM, you wrote: MW> Hi everyone, MW> We (PostgreSQL community) have a HP DL380 G5 that we were using to do MW> some very basic filesystem characterizations as part of a database MW> performance tuning project, so we wanted to give FreeBSD a try out of MW> the box. For this set of data we used 7.1. We're (us few that are MW> running the tests) are fairly unfamiliar with the community here, so MW> I'll be as brief as I can. We're basically wondering if the data MW> we're getting out of the box is expected, and any tuning guidelines MW> including what changes we should expect to see in the performance. -- Best regards, Anthony mailto:ap00@mail.ru From owner-freebsd-performance@FreeBSD.ORG Wed May 6 12:21:31 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31999106566C for ; Wed, 6 May 2009 12:21:31 +0000 (UTC) (envelope-from om-lists-bsd@omx.ch) Received: from ibox.insign.ch (ibox.insign.ch [195.134.143.207]) by mx1.freebsd.org (Postfix) with SMTP id 803438FC1F for ; Wed, 6 May 2009 12:21:30 +0000 (UTC) (envelope-from om-lists-bsd@omx.ch) Received: (qmail 26320 invoked from network); 6 May 2009 11:54:49 -0000 Received: from [192.168.1.170] ([80.254.166.203]) by ibox.insign.ch ([195.134.143.207]) with ESMTP via TCP; 06 May 2009 11:54:49 -0000 From: Olivier Mueller To: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Content-Type: text/plain Date: Wed, 06 May 2009 13:54:48 +0200 Message-Id: <1241610888.16418.64.camel@ompc.insign.local> Mime-Version: 1.0 X-Mailer: Evolution 2.24.1.1 Content-Transfer-Encoding: 7bit Cc: Subject: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 12:21:31 -0000 Hello, $ df -m ; date ; rm -r templates_c ; df -m ; date Filesystem 1M-blocks Used Avail Capacity Mounted on /dev/da0s1a 989 45 864 5% / /dev/da0s1f 128631 102179 16160 86% /usr [...] Wed May 6 00:23:01 CEST 2009 Filesystem 1M-blocks Used Avail Capacity Mounted on /dev/da0s1a 989 45 864 5% / /dev/da0s1f 128631 69844 48496 59% /usr Wed May 6 12:21:02 CEST 2009 -> it took about 12 hours to delete these 30GB of files and sub-directories (smarty cache files: many small files in many dirs). It's a little bit surprising, as it's on a recent HP proliant DL360 g5 with SAS disks (Raid1) running freebsd 6.x ( /dev/da0s1f on /usr (ufs, local, soft-updates) ) Surprisingly, cpu load remained quite low during the operation (apache stayed responsive). Is it a known problem on this kind of hardware or something related to the filesystem? Is there a way to improve this? Even on my $500 PC with IDE disks this goes quicker... :) I checked http://www.freebsd.org/doc/en/books/handbook/configtuning-disk.html but I'm not sure if this would help in this case. Any suggestion how I can "fix" that? Regards, Olivier From owner-freebsd-performance@FreeBSD.ORG Wed May 6 13:08:08 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B519F1065678 for ; Wed, 6 May 2009 13:08:08 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from mail.potentialtech.com (internet.potentialtech.com [66.167.251.6]) by mx1.freebsd.org (Postfix) with ESMTP id 84F798FC1C for ; Wed, 6 May 2009 13:08:08 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from vanquish.ws.pitbpa0.priv.collaborativefusion.com (pr40.pitbpa0.pub.collaborativefusion.com [206.210.89.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.potentialtech.com (Postfix) with ESMTPSA id 29E2CEBC0A; Wed, 6 May 2009 08:48:35 -0400 (EDT) Date: Wed, 6 May 2009 08:48:34 -0400 From: Bill Moran To: Olivier Mueller Message-Id: <20090506084834.61600c42.wmoran@potentialtech.com> In-Reply-To: <1241610888.16418.64.camel@ompc.insign.local> References: <1241610888.16418.64.camel@ompc.insign.local> X-Mailer: Sylpheed 2.6.0 (GTK+ 2.14.7; i386-portbld-freebsd7.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 13:08:09 -0000 In response to Olivier Mueller : > Hello, > > $ df -m ; date ; rm -r templates_c ; df -m ; date > Filesystem 1M-blocks Used Avail Capacity Mounted on > /dev/da0s1a 989 45 864 5% / > /dev/da0s1f 128631 102179 16160 86% /usr > [...] > Wed May 6 00:23:01 CEST 2009 > > Filesystem 1M-blocks Used Avail Capacity Mounted on > /dev/da0s1a 989 45 864 5% / > /dev/da0s1f 128631 69844 48496 59% /usr > Wed May 6 12:21:02 CEST 2009 > > > -> it took about 12 hours to delete these 30GB of files and > sub-directories (smarty cache files: many small files in many dirs). > It's a little bit surprising, as it's on a recent HP proliant DL360 g5 > with SAS disks (Raid1) running freebsd 6.x > ( /dev/da0s1f on /usr (ufs, local, soft-updates) ) > > Surprisingly, cpu load remained quite low during the operation (apache > stayed responsive). Is it a known problem on this kind of hardware or > something related to the filesystem? Is there a way to improve this? > Even on my $500 PC with IDE disks this goes quicker... :) > > I checked > http://www.freebsd.org/doc/en/books/handbook/configtuning-disk.html but > I'm not sure if this would help in this case. Any suggestion how I can > "fix" that? With lots of small files, the time involved is far less dependent on the size of data, and much more dependent on the number of files, and the resultant number of directory entries that need to be updated. "Lots" isn't a particularly accurate count of the # of files, but if you're talking web cache files, I'll guess they average 5k each, which means you had 6 million files. df -i would have been more useful in the output above. This brings a number of questions up: * Are you _sure_ softupdates is enabled on that partition? That's going to make the biggest improvement in speed. * Are these 7200RPM disks or 15,000? Again, going to make a big difference. * If apache was still running, is it possible that it was creating enough disk activity to slow the activity down? Running top -m io will show you how much disk IO each process is creating. * When you compared the speed to your laptop, did you delete 6 million files from the laptop? If you deleted a single 30G file, then you're comparing apples to atom bombs. If this is a directory that you blow away on a regular schedule, you'd do much better to make it a dedicated partition and simply reformat it. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/ From owner-freebsd-performance@FreeBSD.ORG Wed May 6 13:22:04 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0E02106564A for ; Wed, 6 May 2009 13:22:04 +0000 (UTC) (envelope-from om-lists-bsd@omx.ch) Received: from ibox.insign.ch (ibox.insign.ch [195.134.143.207]) by mx1.freebsd.org (Postfix) with SMTP id 70C628FC08 for ; Wed, 6 May 2009 13:22:03 +0000 (UTC) (envelope-from om-lists-bsd@omx.ch) Received: (qmail 11663 invoked from network); 6 May 2009 13:22:02 -0000 Received: from [192.168.1.170] ([80.254.166.203]) by ibox.insign.ch ([195.134.143.207]) with ESMTP via TCP; 06 May 2009 13:22:02 -0000 From: Olivier Mueller To: Bill Moran In-Reply-To: <20090506084834.61600c42.wmoran@potentialtech.com> References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> Content-Type: text/plain Date: Wed, 06 May 2009 15:22:01 +0200 Message-Id: <1241616121.16418.109.camel@ompc.insign.local> Mime-Version: 1.0 X-Mailer: Evolution 2.24.1.1 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data (4 million files) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 13:22:05 -0000 Thanks for your answer Bill! (and to Will as well), Some more infos I gathered a few minutes ago: [~/templates_c]$ date; du -s -m ; date Wed May 6 13:35:15 CEST 2009 2652 . Wed May 6 13:52:36 CEST 2009 [~/templates_c]$ date ; find . | wc -l ; date Wed May 6 13:52:56 CEST 2009 305461 Wed May 6 14:09:39 CEST 2009 So this is on the system after a complete cache cleanup (at 00h00). 300'000 files and 2.6GB. So this night, there were probably around 3-4 million files to delete. Deletion may take time, but 20 minutes juste to _count_ all the files seems pretty long to me... I think I'll say a word to the developers to let them tune their caching system a bit :) On Wed, 2009-05-06 at 08:48 -0400, Bill Moran wrote: > With lots of small files, the time involved is far less dependent on > the size of data, and much more dependent on the number of files, and > the resultant number of directory entries that need to be updated. > "Lots" isn't a particularly accurate count of the # of files, but if > you're talking web cache files, I'll guess they average 5k each, which > means you had 6 million files. df -i would have been more useful in > the output above. Thanks, noted for next time. Now it looks like that: Filesystem 1M-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/da0s1f 128631 70544 47795 60% 1913875 15114219 11% /usr > This brings a number of questions up: > * Are you _sure_ softupdates is enabled on that partition? That's > going to make the biggest improvement in speed. According to "mount" output, yes. I found no specific message about that in the syslog or dmesg. > * Are these 7200RPM disks or 15,000? Again, going to make a big > difference. HP 146GB 6G SAS 10K SFF DP ENT HDD (15k were not available at the time the servers were ordered) ( http://h18004.www1.hp.com/products/servers/proliantstorage/serial/sas/index.html ) > * If apache was still running, is it possible that it was creating > enough disk activity to slow the activity down? Running > top -m io will show you how much disk IO each process is creating. Yes, apache was still running, but the activity was quite low (it was during the night, and the webpage doesn't get so many hits before 9 am local time) While watching "top -m io", the "du" or "find" takes between 80 and 99%, so I guess it's not the probleme here: PID UID VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND 87996 1002 59 56 0 0 0 0 0.00% php 45389 1002 35 25 0 0 2 2 0.84% php 3964 1002 0 0 0 0 0 0 0.00% httpd 3822 1002 151 98 0 0 0 0 0.00% httpd 3005 1002 0 0 0 0 0 0 0.00% httpd 4129 1002 0 0 0 0 0 0 0.00% httpd 3971 1002 0 0 0 0 0 0 0.00% httpd 4231 1002 1 0 0 0 0 0 0.00% httpd 4132 0 234 5 234 0 0 234 97.91% find 98862 1002 1 0 0 0 0 0 0.00% top 609 0 0 0 0 0 0 0 0.00% snmpd [...] > * When you compared the speed to your laptop, did you delete 6 million > files from the laptop? If you deleted a single 30G file, then you're > comparing apples to atom bombs. Yes sorry, I know :) > If this is a directory that you blow away on a regular schedule, you'd > do much better to make it a dedicated partition and simply reformat > it. Yes, it is one of the best options. My initial goal was to delete all files older than N days by cron (find | xargs | rm, etc.), but if each cronjob takes 2 hours (and takes so much cpu time), it's probably not the best way. I'll make some more tests on an test-server later this week and speak with the devs. Thanks again for your very constructive feedback! Regards, Olivier From owner-freebsd-performance@FreeBSD.ORG Wed May 6 13:56:55 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04D091065672 for ; Wed, 6 May 2009 13:56:55 +0000 (UTC) (envelope-from arkadijs.sislovs@affecto.lv) Received: from smtp.affecto.lv (smtp.affecto.lv [213.175.71.131]) by mx1.freebsd.org (Postfix) with ESMTP id B43878FC20 for ; Wed, 6 May 2009 13:56:54 +0000 (UTC) (envelope-from arkadijs.sislovs@affecto.lv) Received: from [10.0.10.50] (helo=hal9000.mebius.lv) by smtp.affecto.lv with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1M1gyP-00049L-4T; Wed, 06 May 2009 16:15:45 +0300 Date: Wed, 6 May 2009 16:15:43 +0300 From: Arkadi Shishlov To: freebsd-performance@freebsd.org Message-ID: <20090506161543.062ba223@hal9000.mebius.lv> In-Reply-To: <20090506084834.61600c42.wmoran@potentialtech.com> References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> Organization: Affecto Latvia X-Mailer: Claws Mail 3.7.1 (GTK+ 2.16.1; i586-redhat-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Wed, 06 May 2009 14:14:14 +0000 Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 13:56:55 -0000 Its probably "dirhash' that is not enabled or its cache is too small for the task. From owner-freebsd-performance@FreeBSD.ORG Wed May 6 14:37:21 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57CB51065676 for ; Wed, 6 May 2009 14:37:21 +0000 (UTC) (envelope-from flo@kasimir.com) Received: from mail.solomo.de (mail.solomo.de [85.214.49.72]) by mx1.freebsd.org (Postfix) with ESMTP id 0F92C8FC0A for ; Wed, 6 May 2009 14:37:20 +0000 (UTC) (envelope-from flo@kasimir.com) Received: from localhost (localhost [127.0.0.1]) by mail.solomo.de (Postfix) with ESMTP id 1BDC13F456; Wed, 6 May 2009 16:18:15 +0200 (CEST) X-Virus-Scanned: amavisd-new at vistream.de Received: from mail.solomo.de ([127.0.0.1]) by localhost (mail.solomo.de [127.0.0.1]) (amavisd-new, port 10024) with LMTP id j4PSOaCLyKcz; Wed, 6 May 2009 16:18:13 +0200 (CEST) Received: from nibbler.vistream.local (relay3.vistream.de [87.139.10.28]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.solomo.de (Postfix) with ESMTPSA id D61DA3F455; Wed, 6 May 2009 16:18:12 +0200 (CEST) Message-ID: <4A019C24.6010804@kasimir.com> Date: Wed, 06 May 2009 16:18:12 +0200 From: Florian Smeets User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b5pre) Gecko/20090505 Shredder/3.0b3pre MIME-Version: 1.0 To: Mark Wong References: <70c01d1d0905042230v3357622cgf4c8e52a2a4ead96@mail.gmail.com> In-Reply-To: <70c01d1d0905042230v3357622cgf4c8e52a2a4ead96@mail.gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org, Selena Deckelmann , Gabrielle Roth Subject: Re: filesystem performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 14:37:21 -0000 On 05.05.09 07:30, Mark Wong wrote: > Hi everyone, > > We (PostgreSQL community) have a HP DL380 G5 that we were using to do > some very basic filesystem characterizations as part of a database > performance tuning project, so we wanted to give FreeBSD a try out of > the box. For this set of data we used 7.1. We're (us few that are > running the tests) are fairly unfamiliar with the community here, so > I'll be as brief as I can. We're basically wondering if the data > we're getting out of the box is expected, and any tuning guidelines > including what changes we should expect to see in the performance. > I guess you are using the ciss driver in this box? There was a performance regression in this driver in 7.1. This should be fixed in 7.2, which came out recently. It is believed that you should get a whole lot better IO performance with 7.2 if you are using the ciss driver. From the 7.2 release notes: A bug in the ciss(4) driver which caused low “max device openings” count and led to poor performance has been fixed. HTH, Florian From owner-freebsd-performance@FreeBSD.ORG Wed May 6 15:01:29 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E17110656A8 for ; Wed, 6 May 2009 15:01:29 +0000 (UTC) (envelope-from markwkm@gmail.com) Received: from mail-gx0-f163.google.com (mail-gx0-f163.google.com [209.85.217.163]) by mx1.freebsd.org (Postfix) with ESMTP id 21B678FC1A for ; Wed, 6 May 2009 15:01:24 +0000 (UTC) (envelope-from markwkm@gmail.com) Received: by gxk7 with SMTP id 7so225944gxk.19 for ; Wed, 06 May 2009 08:01:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=O+8yggfGc80T64v+rH5B1z8Mz1wA4UhbUXteStUByfk=; b=iVCtZNnBCzDNm8iINWfxLFjprgjWNsGTGTnw1GM6Sm2H+2e4egRFNpn6qoI1oLSR87 aCb0sJnZ/cVSYcJHDS4kRCMLyQv9dKyB5i1D7eXwL0GCNxgkc4jVYX7xnwQmYMWY9otP N1BKjHpRAplXOFGRY+frkXHPz7+rbDJY5iK8k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=nXdwucvX0FfMI7juFhUtXPq73Ftfsj/QcR2u0CHICZWq3UjZs5Q4a5xx6BUktj2fAG A/OgxZ8L6a91UyqgipMZxeg/8fawKqeYjmriCD8RiLW88D6ydsF4kaqStgZoAe1/RGY1 vAi5o/fvqr3vf8kt6OZY1VielerLHN5ePhavc= MIME-Version: 1.0 Received: by 10.100.251.8 with SMTP id y8mr3043368anh.74.1241622073334; Wed, 06 May 2009 08:01:13 -0700 (PDT) In-Reply-To: <5871156390.20090506132550@mail.ru> References: <70c01d1d0905042230v3357622cgf4c8e52a2a4ead96@mail.gmail.com> <5871156390.20090506132550@mail.ru> Date: Wed, 6 May 2009 08:01:13 -0700 Message-ID: <70c01d1d0905060801r1eb7b9f7o5c1c9505130a7667@mail.gmail.com> From: Mark Wong To: Anthony Pankov Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org, Selena Deckelmann , Gabrielle Roth Subject: Re: filesystem performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 15:01:30 -0000 On Wed, May 6, 2009 at 2:25 AM, Anthony Pankov wrote: > Hello Mark, > > May i ask a question while more expierenced people is waking up? > > I don't fully understand the target. For what filesystem should be > optimized? > > I expect a patterns of recorded IO calls when pgsql perform typical > operations with statistics and in-depth analysis. The angle we're trying to look at is from a sizing perspective. In order words we want to have an idea of what to expect before we do it. For example, if I have 10 drives, what can I expect if I configure them in a RAID 10 configuration? > Are you sure there is =B1 strong relation between fio benchmark result an= d > PostgreSQL performance? Sorry, this was something I was trying to make clearer originally. No, I don't think there is a strong relationship between fio and PostgreSQL, but these i/o patterns we are simulating do give us a rough estimate of we can expect. For example, in workloads with lots of update and inserts into a database will generate a lot of sequential writes to the database logs, which we can physically isolate onto it's own lun. Similarly, in some warehousing applications there may be a table that is always scanned and read sequential, which also can be on its own physical lun. Regards, Mark From owner-freebsd-performance@FreeBSD.ORG Wed May 6 15:02:34 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 989521065679 for ; Wed, 6 May 2009 15:02:34 +0000 (UTC) (envelope-from om-lists-bsd@omx.ch) Received: from ibox.insign.ch (ibox.insign.ch [195.134.143.207]) by mx1.freebsd.org (Postfix) with SMTP id E21238FC16 for ; Wed, 6 May 2009 15:02:28 +0000 (UTC) (envelope-from om-lists-bsd@omx.ch) Received: (qmail 7154 invoked from network); 6 May 2009 15:02:27 -0000 Received: from [192.168.1.170] ([80.254.166.203]) by ibox.insign.ch ([195.134.143.207]) with ESMTP via TCP; 06 May 2009 15:02:27 -0000 From: Olivier Mueller To: Arkadi Shishlov In-Reply-To: <20090506161543.062ba223@hal9000.mebius.lv> References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <20090506161543.062ba223@hal9000.mebius.lv> Content-Type: text/plain Date: Wed, 06 May 2009 17:02:26 +0200 Message-Id: <1241622146.16418.128.camel@ompc.insign.local> Mime-Version: 1.0 X-Mailer: Evolution 2.24.1.1 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 15:02:35 -0000 On Wed, 2009-05-06 at 16:15 +0300, Arkadi Shishlov wrote: > Its probably "dirhash' that is not enabled or its cache is too small for the task. $ sysctl -a |grep dirha UFS dirhash 1262 286K - 9715683 16,32,64,128,256,512,1024,2048,4096 vfs.ufs.dirhash_docheck: 0 vfs.ufs.dirhash_mem: 2087495 vfs.ufs.dirhash_maxmem: 2097152 vfs.ufs.dirhash_minsize: 2560 So it's active, but probably too small as you suggest. Can I update this value "on the fly" or does it require a reboot (+ settings in loader.conf) ? regards, Olivier From owner-freebsd-performance@FreeBSD.ORG Wed May 6 15:03:01 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0DE6F10656CA for ; Wed, 6 May 2009 15:03:01 +0000 (UTC) (envelope-from markwkm@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.28]) by mx1.freebsd.org (Postfix) with ESMTP id B978C8FC1F for ; Wed, 6 May 2009 15:03:00 +0000 (UTC) (envelope-from markwkm@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so82537yxb.13 for ; Wed, 06 May 2009 08:03:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=4mroSo0i1g5nDK7VkOQCoFPUjMHotzKBr7kQWlojmZA=; b=YhYu9uelBnitq4m2+Ku2Wcm8d8yMoaPjCA24xln77NY2J1xUMZiNAhzkITcurT/YFv rqBA+uMC4UYP3VFmvLj070ioNhsL5P+ztXsIZwApz5m4x0AM1TcMBsN4qRzdRaS1sUei NoFEp5Qx65n1fSZhs6lxsGPC9n6KrxlkXjRuQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=n06LCmGWwQvEaQ15x9WfFh1VMTsEpwDm0d/H0kSFsy3LnSWd6rqPfWTU7Z+86s+3YQ pqfsSmj/fxYZvsX2fQAF/Sy30B/rtSR7Wdg+Ba2kLhEstKRmdFzR5h2nAo2yyhRwke0F EKR23iBX8GYxzn8uhRQgBkgSSWPHrEXPC4F3Y= MIME-Version: 1.0 Received: by 10.100.178.9 with SMTP id a9mr3117848anf.11.1241622179972; Wed, 06 May 2009 08:02:59 -0700 (PDT) In-Reply-To: <4A019C24.6010804@kasimir.com> References: <70c01d1d0905042230v3357622cgf4c8e52a2a4ead96@mail.gmail.com> <4A019C24.6010804@kasimir.com> Date: Wed, 6 May 2009 08:02:59 -0700 Message-ID: <70c01d1d0905060802y3eeb80c7m59451d29d99ae7cc@mail.gmail.com> From: Mark Wong To: Florian Smeets Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org, Selena Deckelmann , Gabrielle Roth Subject: Re: filesystem performance X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 15:03:01 -0000 On Wed, May 6, 2009 at 7:18 AM, Florian Smeets wrote: > On 05.05.09 07:30, Mark Wong wrote: >> >> Hi everyone, >> >> We (PostgreSQL community) have a HP DL380 G5 that we were using to do >> some very basic filesystem characterizations as part of a database >> performance tuning project, so we wanted to give FreeBSD a try out of >> the box. =A0For this set of data we used 7.1. =A0We're (us few that are >> running the tests) are fairly unfamiliar with the community here, so >> I'll be as brief as I can. =A0We're basically wondering if the data >> we're getting out of the box is expected, and any tuning guidelines >> including what changes we should expect to see in the performance. >> > > I guess you are using the ciss driver in this box? There was a performanc= e > regression in this driver in 7.1. This should be fixed in 7.2, which came > out recently. It is believed that you should get a whole lot better IO > performance with 7.2 if you are using the ciss driver. > > From the 7.2 release notes: > > A bug in the ciss(4) driver which caused low =93max device openings=94 co= unt and > led to poor performance has been fixed. We'll have to make time to try that. :) Thanks (and to the others) for pointing that out. Regards, Mark From owner-freebsd-performance@FreeBSD.ORG Wed May 6 15:05:54 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 865361065679 for ; Wed, 6 May 2009 15:05:54 +0000 (UTC) (envelope-from wmoran@collaborativefusion.com) Received: from mx00.pub.collaborativefusion.com (mx00.pub.collaborativefusion.com [206.210.89.199]) by mx1.freebsd.org (Postfix) with ESMTP id 314938FC24 for ; Wed, 6 May 2009 15:05:54 +0000 (UTC) (envelope-from wmoran@collaborativefusion.com) Received: from vanquish.ws.pitbpa0.priv.collaborativefusion.com (vanquish.ws.pitbpa0.priv.collaborativefusion.com [192.168.2.162]) (SSL: TLSv1/SSLv3,256bits,AES256-SHA) by wingspan with esmtp; Wed, 06 May 2009 10:55:42 -0400 id 0005642B.000000004A01A4EE.00015A47 Date: Wed, 6 May 2009 10:55:42 -0400 From: Bill Moran To: Arkadi Shishlov Message-Id: <20090506105542.45212e34.wmoran@collaborativefusion.com> In-Reply-To: <20090506161543.062ba223@hal9000.mebius.lv> References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <20090506161543.062ba223@hal9000.mebius.lv> Organization: Collaborative Fusion Inc. X-Mailer: Sylpheed 2.6.0 (GTK+ 2.14.7; i386-portbld-freebsd7.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 15:05:54 -0000 In response to Arkadi Shishlov : > Its probably "dirhash' that is not enabled or its cache is too small for the task. I'm no expert, but I thought dirhash only improved read speed. His bottleneck would be writes. -- Bill Moran Collaborative Fusion Inc. http://people.collaborativefusion.com/~wmoran/ wmoran@collaborativefusion.com Phone: 412-422-3463x4023 **************************************************************** IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. **************************************************************** From owner-freebsd-performance@FreeBSD.ORG Wed May 6 15:10:24 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB2AD106571E; Wed, 6 May 2009 15:10:24 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from mail.potentialtech.com (internet.potentialtech.com [66.167.251.6]) by mx1.freebsd.org (Postfix) with ESMTP id 8A8188FC12; Wed, 6 May 2009 15:10:24 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from vanquish.ws.pitbpa0.priv.collaborativefusion.com (pr40.pitbpa0.pub.collaborativefusion.com [206.210.89.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.potentialtech.com (Postfix) with ESMTPSA id 14935EBC0A; Wed, 6 May 2009 11:10:23 -0400 (EDT) Date: Wed, 6 May 2009 11:10:22 -0400 From: Bill Moran To: Olivier Mueller Message-Id: <20090506111022.05d06f1a.wmoran@potentialtech.com> In-Reply-To: <1241616121.16418.109.camel@ompc.insign.local> References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <1241616121.16418.109.camel@ompc.insign.local> X-Mailer: Sylpheed 2.6.0 (GTK+ 2.14.7; i386-portbld-freebsd7.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data (4 million files) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 15:10:25 -0000 In response to Olivier Mueller : > > Yes, it is one of the best options. My initial goal was to delete all > files older than N days by cron (find | xargs | rm, etc.), but if each > cronjob takes 2 hours (and takes so much cpu time), it's probably not > the best way. > > I'll make some more tests on an test-server later this week and speak > with the devs. Thanks again for your very constructive feedback! Based on your comments here, it really sounds like your devs need to implement some sort of cache cleaning algo into their code. If it's just deleting the oldest files, then you could probably run it far more frequently if you simply created a new cache directory each hour, and deleted the previous one. Honestly, I'm really confused -- if you can just throw away the cache each night, then why are you caching to begin with? If you just need temp files, why doesn't the app clean up its temp files when it's done with them? If you have access to the developers, I think you'll be able to come up with a much better solution by working with them. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/ From owner-freebsd-performance@FreeBSD.ORG Wed May 6 15:50:49 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5F0231065718; Wed, 6 May 2009 15:50:49 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [IPv6:2001:4070:101:2::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5848D8FC23; Wed, 6 May 2009 15:50:47 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (localhost [IPv6:::1]) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3) with ESMTP id n46FnrAs032788; Wed, 6 May 2009 17:49:55 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3/Submit) with ESMTP id n46FYOgk032592; Wed, 6 May 2009 17:34:25 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Wed, 6 May 2009 17:34:24 +0200 (CEST) From: Wojciech Puchar To: Olivier Mueller In-Reply-To: <1241610888.16418.64.camel@ompc.insign.local> Message-ID: References: <1241610888.16418.64.camel@ompc.insign.local> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Wed, 06 May 2009 16:01:16 +0000 Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 15:50:50 -0000 > -> it took about 12 hours to delete these 30GB of files and > sub-directories (smarty cache files: many small files in many dirs). > It's a little bit surprising, as it's on a recent HP proliant DL360 g5 > with SAS disks (Raid1) running freebsd 6.x > ( /dev/da0s1f on /usr (ufs, local, soft-updates) ) > if you would use no raid or software raid it will behave normally. it takes <30 minutes for me to delete 300GB of squid files on ordinary SATA disk , millions of small files. From owner-freebsd-performance@FreeBSD.ORG Wed May 6 15:50:50 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7814F1065719; Wed, 6 May 2009 15:50:49 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [IPv6:2001:4070:101:2::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5854C8FC2F; Wed, 6 May 2009 15:50:47 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (localhost [IPv6:::1]) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3) with ESMTP id n46FnrAu032788; Wed, 6 May 2009 17:49:55 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3/Submit) with ESMTP id n46FaBfN032593; Wed, 6 May 2009 17:36:11 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Wed, 6 May 2009 17:36:11 +0200 (CEST) From: Wojciech Puchar To: Bill Moran In-Reply-To: <20090506084834.61600c42.wmoran@potentialtech.com> Message-ID: References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Wed, 06 May 2009 16:01:27 +0000 Cc: freebsd-performance@freebsd.org, Olivier Mueller , freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 15:50:51 -0000 > means you had 6 million files. df -i would have been more useful in > the output above. > > This brings a number of questions up: > * Are you _sure_ softupdates is enabled on that partition? That's he showed mount output - he has softdeps on. > * Are these 7200RPM disks or 15,000? Again, going to make a big > difference. on 7200 RPM ordinary SATA disk i deleted 15 million files taking 300GB (squid cache) in less than 30 minutes. for sure it's because of his "hardware raid". i've NEVER seen "hardware raid" that is actually faster than non-raid config, or gmirror/gstripe config. usually it's far much slower From owner-freebsd-performance@FreeBSD.ORG Wed May 6 17:16:20 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F306106566C; Wed, 6 May 2009 17:16:20 +0000 (UTC) (envelope-from benjamin@seattlefenix.net) Received: from mx1.seattlefenix.net (mx1.seattlefenix.net [208.75.57.33]) by mx1.freebsd.org (Postfix) with ESMTP id 73CA88FC14; Wed, 6 May 2009 17:16:20 +0000 (UTC) (envelope-from benjamin@seattlefenix.net) Received: from [10.0.0.228] (unknown [64.81.172.79]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.seattlefenix.net (Postfix) with ESMTPSA id 5E9633F69D; Wed, 6 May 2009 10:00:10 -0700 (PDT) Message-ID: <4A01C202.8080803@seattlefenix.net> Date: Wed, 06 May 2009 09:59:46 -0700 From: Benjamin Krueger User-Agent: Thunderbird 2.0.0.21 (X11/20090318) MIME-Version: 1.0 To: Wojciech Puchar References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Olivier Mueller , Bill Moran , freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 17:16:21 -0000 Wojciech Puchar wrote: >> means you had 6 million files. df -i would have been more useful in > > >> the output above. >> >> This brings a number of questions up: >> * Are you _sure_ softupdates is enabled on that partition? That's > > he showed mount output - he has softdeps on. > >> * Are these 7200RPM disks or 15,000? Again, going to make a big >> difference. > > on 7200 RPM ordinary SATA disk i deleted 15 million files taking 300GB > (squid cache) in less than 30 minutes. > > for sure it's because of his "hardware raid". > > i've NEVER seen "hardware raid" that is actually faster than non-raid > config, or gmirror/gstripe config. > > usually it's far much slower Sorry, but my experience with that very server using a P400 controller with 256MB write cache is very different. My benchmarks showed that controller using Raid5 (with only 4 disks) is significantly faster than software layouts. The days when hardware controllers could automatically be considered slow are long gone. The hardware does get faster over time. Don't make any assumptions without doing benchmarks. From owner-freebsd-performance@FreeBSD.ORG Wed May 6 18:29:54 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35F591065673; Wed, 6 May 2009 18:29:54 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from mail.potentialtech.com (internet.potentialtech.com [66.167.251.6]) by mx1.freebsd.org (Postfix) with ESMTP id 027228FC1C; Wed, 6 May 2009 18:29:53 +0000 (UTC) (envelope-from wmoran@potentialtech.com) Received: from vanquish.ws.pitbpa0.priv.collaborativefusion.com (pr40.pitbpa0.pub.collaborativefusion.com [206.210.89.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.potentialtech.com (Postfix) with ESMTPSA id 29BA0EBC0A; Wed, 6 May 2009 14:29:53 -0400 (EDT) Date: Wed, 6 May 2009 14:29:51 -0400 From: Bill Moran To: "Gary Gatten" Message-Id: <20090506142951.2a27284d.wmoran@potentialtech.com> In-Reply-To: <70C0964126D66F458E688618E1CD008A0793EBD1@WADPEXV0.waddell.com> References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <4A01C202.8080803@seattlefenix.net> <70C0964126D66F458E688618E1CD008A0793EBD1@WADPEXV0.waddell.com> X-Mailer: Sylpheed 2.6.0 (GTK+ 2.14.7; i386-portbld-freebsd7.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: Mueller , Benjamin Krueger , Olivier, Wojciech Puchar , freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 18:29:54 -0000 In response to "Gary Gatten" : > It could just be me, but I swear Hardware RAID has been faster for many > many years, especially with RAID5 arrays - or anything that requires > parity calcs. Most of my benchmarking was done on SCO OpenServer and > Novell UnixWare and Netware, but hardware RAID controllers were always > faster and of course required far less host CPU resources. Raid > 0/1/10/0+1/whatever arrays, I recall weren't as drastic, but I can't > imagine the controller making as big a difference as the drives in the > array - unless of course the drive for said controller sux! Keep in mind that there are a LOT of RAID controllers out there, and yes, some of them suck royally. Especially the consumer-grade stuff intended for people to use on their home systems. I'd be willing to bet that software RAID is faster than 90% of the consumer grade RAID cards, and probably more reliable than most of them as well. Controllers make a huge difference, even in server class RAID (in my experience). There is a significant gap in performance between the good stuff and the good enough stuff. -- Bill Moran http://www.potentialtech.com http://people.collaborativefusion.com/~wmoran/ From owner-freebsd-performance@FreeBSD.ORG Wed May 6 18:31:06 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7871B1065672; Wed, 6 May 2009 18:31:06 +0000 (UTC) (envelope-from Ggatten@waddell.com) Received: from mailhost0.waddell.com (mailhost0.waddell.com [12.154.38.61]) by mx1.freebsd.org (Postfix) with ESMTP id 433CA8FC1B; Wed, 6 May 2009 18:31:05 +0000 (UTC) (envelope-from Ggatten@waddell.com) Received: from mailhost2.waddell.com (mailhost2.waddell.com [10.1.10.30]) by mailhost0.waddell.com (8.13.8/8.13.8) with ESMTP id n46IUUvo002033; Wed, 6 May 2009 13:30:40 -0500 (CDT) (envelope-from Ggatten@waddell.com) Received: from mailhost2.waddell.com (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 8D90B8A4DD; Wed, 6 May 2009 13:30:29 -0500 (CDT) Received: from wadpexf0.waddell.com (wadpexf0.waddell.com [192.168.204.24]) by mailhost2.waddell.com (Postfix) with ESMTP id 5672A8A4B5; Wed, 6 May 2009 13:30:29 -0500 (CDT) Received: from WADPEXV0.waddell.com ([192.168.204.25]) by wadpexf0.waddell.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 6 May 2009 13:30:28 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Wed, 6 May 2009 13:30:01 -0500 Message-ID: <70C0964126D66F458E688618E1CD008A0793EBD2@WADPEXV0.waddell.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: filesystem: 12h to delete 32GB of data Thread-Index: AcnObqfHTigM1AG/QCKN2todI36EswABn39AAADhS5Q= From: "Gary Gatten" To: , X-OriginalArrivalTime: 06 May 2009 18:30:28.0942 (UTC) FILETIME=[BB3BB6E0:01C9CE78] X-Mailman-Approved-At: Wed, 06 May 2009 18:50:18 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, om-lists-bsd@omx.ch, wmoran@potentialtech.com, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 18:31:07 -0000 U29ycnksICJkcml2ZSIgaW4gbGFzdCBzZW50ZW5jZSBzaG91bGQgYmUgImRyaXZlciIhDQoNCi0t LS0tIE9yaWdpbmFsIE1lc3NhZ2UgLS0tLS0NCkZyb206IG93bmVyLWZyZWVic2QtcXVlc3Rpb25z QGZyZWVic2Qub3JnIDxvd25lci1mcmVlYnNkLXF1ZXN0aW9uc0BmcmVlYnNkLm9yZz4NClRvOiBC ZW5qYW1pbiBLcnVlZ2VyIDxiZW5qYW1pbkBzZWF0dGxlZmVuaXgubmV0PjsgV29qY2llY2ggUHVj aGFyIDx3b2p0ZWtAd29qdGVrLnRlbnNvci5nZHluaWEucGw+DQpDYzogZnJlZWJzZC1wZXJmb3Jt YW5jZUBmcmVlYnNkLm9yZyA8ZnJlZWJzZC1wZXJmb3JtYW5jZUBmcmVlYnNkLm9yZz47IE9saXZp ZXIgTXVlbGxlciA8b20tbGlzdHMtYnNkQG9teC5jaD47IEJpbGwgTW9yYW4gPHdtb3JhbkBwb3Rl bnRpYWx0ZWNoLmNvbT47IGZyZWVic2QtcXVlc3Rpb25zQGZyZWVic2Qub3JnIDxmcmVlYnNkLXF1 ZXN0aW9uc0BmcmVlYnNkLm9yZz4NClNlbnQ6IFdlZCBNYXkgMDYgMTM6MDg6NDYgMjAwOQ0KU3Vi amVjdDogUkU6IGZpbGVzeXN0ZW06IDEyaCB0byBkZWxldGUgMzJHQiBvZiBkYXRhDQoNCkl0IGNv dWxkIGp1c3QgYmUgbWUsIGJ1dCBJIHN3ZWFyIEhhcmR3YXJlIFJBSUQgaGFzIGJlZW4gZmFzdGVy IGZvciBtYW55DQptYW55IHllYXJzLCBlc3BlY2lhbGx5IHdpdGggUkFJRDUgYXJyYXlzIC0gb3Ig YW55dGhpbmcgdGhhdCByZXF1aXJlcw0KcGFyaXR5IGNhbGNzLiAgTW9zdCBvZiBteSBiZW5jaG1h cmtpbmcgd2FzIGRvbmUgb24gU0NPIE9wZW5TZXJ2ZXIgYW5kDQpOb3ZlbGwgVW5peFdhcmUgYW5k IE5ldHdhcmUsIGJ1dCBoYXJkd2FyZSBSQUlEIGNvbnRyb2xsZXJzIHdlcmUgYWx3YXlzDQpmYXN0 ZXIgYW5kIG9mIGNvdXJzZSByZXF1aXJlZCBmYXIgbGVzcyBob3N0IENQVSByZXNvdXJjZXMuICBS YWlkDQowLzEvMTAvMCsxL3doYXRldmVyIGFycmF5cywgSSByZWNhbGwgd2VyZW4ndCBhcyBkcmFz dGljLCBidXQgSSBjYW4ndA0KaW1hZ2luZSB0aGUgY29udHJvbGxlciBtYWtpbmcgYXMgYmlnIGEg ZGlmZmVyZW5jZSBhcyB0aGUgZHJpdmVzIGluIHRoZQ0KYXJyYXkgLSB1bmxlc3Mgb2YgY291cnNl IHRoZSBkcml2ZSBmb3Igc2FpZCBjb250cm9sbGVyIHN1eCENCg0KPHNuaXA+DQoNCg0KDQoNCg0K DQo8Zm9udCBzaXplPSIxIj4NCjxkaXYgc3R5bGU9J2JvcmRlcjpub25lO2JvcmRlci1ib3R0b206 ZG91YmxlIHdpbmRvd3RleHQgMi4yNXB0O3BhZGRpbmc6MGluIDBpbiAxLjBwdCAwaW4nPg0KPC9k aXY+DQoiVGhpcyBlbWFpbCBpcyBpbnRlbmRlZCB0byBiZSByZXZpZXdlZCBieSBvbmx5IHRoZSBp bnRlbmRlZCByZWNpcGllbnQNCiBhbmQgbWF5IGNvbnRhaW4gaW5mb3JtYXRpb24gdGhhdCBpcyBw cml2aWxlZ2VkIGFuZC9vciBjb25maWRlbnRpYWwuDQogSWYgeW91IGFyZSBub3QgdGhlIGludGVu ZGVkIHJlY2lwaWVudCwgeW91IGFyZSBoZXJlYnkgbm90aWZpZWQgdGhhdA0KIGFueSByZXZpZXcs IHVzZSwgZGlzc2VtaW5hdGlvbiwgZGlzY2xvc3VyZSBvciBjb3B5aW5nIG9mIHRoaXMgZW1haWwN CiBhbmQgaXRzIGF0dGFjaG1lbnRzLCBpZiBhbnksIGlzIHN0cmljdGx5IHByb2hpYml0ZWQuICBJ ZiB5b3UgaGF2ZQ0KIHJlY2VpdmVkIHRoaXMgZW1haWwgaW4gZXJyb3IsIHBsZWFzZSBpbW1lZGlh dGVseSBub3RpZnkgdGhlIHNlbmRlciBieQ0KIHJldHVybiBlbWFpbCBhbmQgZGVsZXRlIHRoaXMg ZW1haWwgZnJvbSB5b3VyIHN5c3RlbS4iDQo8L2ZvbnQ+DQoNCl9fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fDQpmcmVlYnNkLXF1ZXN0aW9uc0BmcmVlYnNkLm9y ZyBtYWlsaW5nIGxpc3QNCmh0dHA6Ly9saXN0cy5mcmVlYnNkLm9yZy9tYWlsbWFuL2xpc3RpbmZv L2ZyZWVic2QtcXVlc3Rpb25zDQpUbyB1bnN1YnNjcmliZSwgc2VuZCBhbnkgbWFpbCB0byAiZnJl ZWJzZC1xdWVzdGlvbnMtdW5zdWJzY3JpYmVAZnJlZWJzZC5vcmciDQoKCgoKCjxmb250IHNpemU9 IjEiPgo8ZGl2IHN0eWxlPSdib3JkZXI6bm9uZTtib3JkZXItYm90dG9tOmRvdWJsZSB3aW5kb3d0 ZXh0IDIuMjVwdDtwYWRkaW5nOjBpbiAwaW4gMS4wcHQgMGluJz4KPC9kaXY+CiJUaGlzIGVtYWls IGlzIGludGVuZGVkIHRvIGJlIHJldmlld2VkIGJ5IG9ubHkgdGhlIGludGVuZGVkIHJlY2lwaWVu dAogYW5kIG1heSBjb250YWluIGluZm9ybWF0aW9uIHRoYXQgaXMgcHJpdmlsZWdlZCBhbmQvb3Ig Y29uZmlkZW50aWFsLgogSWYgeW91IGFyZSBub3QgdGhlIGludGVuZGVkIHJlY2lwaWVudCwgeW91 IGFyZSBoZXJlYnkgbm90aWZpZWQgdGhhdAogYW55IHJldmlldywgdXNlLCBkaXNzZW1pbmF0aW9u LCBkaXNjbG9zdXJlIG9yIGNvcHlpbmcgb2YgdGhpcyBlbWFpbAogYW5kIGl0cyBhdHRhY2htZW50 cywgaWYgYW55LCBpcyBzdHJpY3RseSBwcm9oaWJpdGVkLiAgSWYgeW91IGhhdmUKIHJlY2VpdmVk IHRoaXMgZW1haWwgaW4gZXJyb3IsIHBsZWFzZSBpbW1lZGlhdGVseSBub3RpZnkgdGhlIHNlbmRl ciBieQogcmV0dXJuIGVtYWlsIGFuZCBkZWxldGUgdGhpcyBlbWFpbCBmcm9tIHlvdXIgc3lzdGVt LiIKPC9mb250PgoK From owner-freebsd-performance@FreeBSD.ORG Wed May 6 18:31:13 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71738106564A; Wed, 6 May 2009 18:31:13 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [IPv6:2001:4070:101:2::1]) by mx1.freebsd.org (Postfix) with ESMTP id 741738FC13; Wed, 6 May 2009 18:31:10 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (localhost [IPv6:::1]) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3) with ESMTP id n46IUIVG034027; Wed, 6 May 2009 20:30:18 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3/Submit) with ESMTP id n46IUH2R034024; Wed, 6 May 2009 20:30:17 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Wed, 6 May 2009 20:30:17 +0200 (CEST) From: Wojciech Puchar To: Benjamin Krueger In-Reply-To: <4A01C202.8080803@seattlefenix.net> Message-ID: References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <4A01C202.8080803@seattlefenix.net> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Wed, 06 May 2009 18:50:30 +0000 Cc: freebsd-performance@freebsd.org, Olivier Mueller , Bill Moran , freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 18:31:14 -0000 >> config, or gmirror/gstripe config. >> >> usually it's far much slower > > Sorry, but my experience with that very server using a P400 controller with > 256MB write cache is very different. My benchmarks showed that controller > using Raid5 (with only 4 disks) is significantly faster than software > layouts. possibly with RAID5, but for sure slower than single drive > The days when hardware controllers could automatically be considered slow are > long gone. unfortunately not. From owner-freebsd-performance@FreeBSD.ORG Wed May 6 18:32:11 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 843A51065676; Wed, 6 May 2009 18:32:11 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [IPv6:2001:4070:101:2::1]) by mx1.freebsd.org (Postfix) with ESMTP id ED5A48FC0A; Wed, 6 May 2009 18:32:08 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (localhost [IPv6:::1]) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3) with ESMTP id n46IVGYR034047; Wed, 6 May 2009 20:31:16 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3/Submit) with ESMTP id n46IVGx9034044; Wed, 6 May 2009 20:31:16 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Wed, 6 May 2009 20:31:16 +0200 (CEST) From: Wojciech Puchar To: Gary Gatten In-Reply-To: <70C0964126D66F458E688618E1CD008A0793EBD1@WADPEXV0.waddell.com> Message-ID: References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <4A01C202.8080803@seattlefenix.net> <70C0964126D66F458E688618E1CD008A0793EBD1@WADPEXV0.waddell.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Wed, 06 May 2009 18:50:40 +0000 Cc: freebsd-questions@freebsd.org, Benjamin Krueger , Olivier Mueller , freebsd-performance@freebsd.org, Bill Moran Subject: RE: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 18:32:12 -0000 > It could just be me, but I swear Hardware RAID has been faster for many > many years, especially with RAID5 arrays - or anything that requires maybe with RAID5, but using RAID5 today (huge disk sizes, little sense to save on disk space) instead of RAID1/10 doesn't make much sense, as RAID5 is slow on writes by design From owner-freebsd-performance@FreeBSD.ORG Wed May 6 18:32:46 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5DE9E10656DF; Wed, 6 May 2009 18:32:46 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [IPv6:2001:4070:101:2::1]) by mx1.freebsd.org (Postfix) with ESMTP id 91B5D8FC17; Wed, 6 May 2009 18:32:45 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (localhost [IPv6:::1]) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3) with ESMTP id n46IVsx2034077; Wed, 6 May 2009 20:31:54 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3/Submit) with ESMTP id n46IVshV034074; Wed, 6 May 2009 20:31:54 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Wed, 6 May 2009 20:31:53 +0200 (CEST) From: Wojciech Puchar To: Bill Moran In-Reply-To: <20090506142951.2a27284d.wmoran@potentialtech.com> Message-ID: References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <4A01C202.8080803@seattlefenix.net> <70C0964126D66F458E688618E1CD008A0793EBD1@WADPEXV0.waddell.com> <20090506142951.2a27284d.wmoran@potentialtech.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Wed, 06 May 2009 18:50:51 +0000 Cc: Gary Gatten , Benjamin Krueger , Olivier Mueller , freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 18:32:47 -0000 > yes, some of them suck royally. you should rather say "some of them doesn't suck". From owner-freebsd-performance@FreeBSD.ORG Wed May 6 18:50:23 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E0AA1065670; Wed, 6 May 2009 18:50:23 +0000 (UTC) (envelope-from Ggatten@waddell.com) Received: from mailhost0.waddell.com (mailhost0.waddell.com [12.154.38.61]) by mx1.freebsd.org (Postfix) with ESMTP id DE72B8FC27; Wed, 6 May 2009 18:50:17 +0000 (UTC) (envelope-from Ggatten@waddell.com) Received: from mailhost2.waddell.com (mailhost2.waddell.com [10.1.10.30]) by mailhost0.waddell.com (8.13.8/8.13.8) with ESMTP id n46Io6v1018552; Wed, 6 May 2009 13:50:12 -0500 (CDT) (envelope-from Ggatten@waddell.com) Received: from mailhost2.waddell.com (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 47FCA8A4CB; Wed, 6 May 2009 13:50:06 -0500 (CDT) Received: from wadpexf0.waddell.com (wadpexf0.waddell.com [192.168.204.24]) by mailhost2.waddell.com (Postfix) with ESMTP id 076378A4F9; Wed, 6 May 2009 13:50:06 -0500 (CDT) Received: from WADPEXV0.waddell.com ([192.168.204.25]) by wadpexf0.waddell.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 6 May 2009 13:50:05 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Wed, 6 May 2009 13:49:41 -0500 Message-ID: <70C0964126D66F458E688618E1CD008A0793EBD4@WADPEXV0.waddell.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: filesystem: 12h to delete 32GB of data Thread-Index: AcnOeUGn30arYdYkS6CBU725GiG7YQAAihvL From: "Gary Gatten" X-OriginalArrivalTime: 06 May 2009 18:50:05.0669 (UTC) FILETIME=[789E0150:01C9CE7B] X-Mailman-Approved-At: Wed, 06 May 2009 19:06:53 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 18:50:23 -0000 T1Qgbm93LCBidXQgaW4gaGlnaCBpL28gZW52cyB3aXRoIGhpZ2ggY29uY3VycmVuY3kgbmVlZHMs IFJBSUQ1IGlzIHN0aWxsIHRoZSB3YXkgdG8gZ28sIGVzcCBpZiA5MCUgb2YgaS9vIGlzIHJlYWRz LiBPZiBjb3Vyc2UgaXQgZGVwZW5kcyBvbiBmaWxlIHNpemUgLyB0eXBlIGFzIHdlbGwuLi4gQW55 d2F5LCBsZXQncyBzdW0gaXQgdXAgd2l0aCAiYSBzdG9yYWdlIHN1YnN5c3RlbSBpcyBvbmx5IGFz IGZhc3QgYXMgaXRzIHNsb3dlc3QgbGluayINCg0KLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0t LQ0KRnJvbTogV29qY2llY2ggUHVjaGFyIDx3b2p0ZWtAd29qdGVrLnRlbnNvci5nZHluaWEucGw+ DQpUbzogQmlsbCBNb3JhbiA8d21vcmFuQHBvdGVudGlhbHRlY2guY29tPg0KQ2M6IEdhcnkgR2F0 dGVuOyBCZW5qYW1pbiBLcnVlZ2VyIDxiZW5qYW1pbkBzZWF0dGxlZmVuaXgubmV0PjsgZnJlZWJz ZC1wZXJmb3JtYW5jZUBmcmVlYnNkLm9yZyA8ZnJlZWJzZC1wZXJmb3JtYW5jZUBmcmVlYnNkLm9y Zz47IE9saXZpZXIgTXVlbGxlciA8b20tbGlzdHMtYnNkQG9teC5jaD47IGZyZWVic2QtcXVlc3Rp b25zQGZyZWVic2Qub3JnIDxmcmVlYnNkLXF1ZXN0aW9uc0BmcmVlYnNkLm9yZz4NClNlbnQ6IFdl ZCBNYXkgMDYgMTM6MzE6NTMgMjAwOQ0KU3ViamVjdDogUmU6IGZpbGVzeXN0ZW06IDEyaCB0byBk ZWxldGUgMzJHQiBvZiBkYXRhDQoNCj4geWVzLCBzb21lIG9mIHRoZW0gc3VjayByb3lhbGx5Lg0K DQp5b3Ugc2hvdWxkIHJhdGhlciBzYXkgInNvbWUgb2YgdGhlbSBkb2Vzbid0IHN1Y2siLg0KCgoK Cgo8Zm9udCBzaXplPSIxIj4KPGRpdiBzdHlsZT0nYm9yZGVyOm5vbmU7Ym9yZGVyLWJvdHRvbTpk b3VibGUgd2luZG93dGV4dCAyLjI1cHQ7cGFkZGluZzowaW4gMGluIDEuMHB0IDBpbic+CjwvZGl2 PgoiVGhpcyBlbWFpbCBpcyBpbnRlbmRlZCB0byBiZSByZXZpZXdlZCBieSBvbmx5IHRoZSBpbnRl bmRlZCByZWNpcGllbnQKIGFuZCBtYXkgY29udGFpbiBpbmZvcm1hdGlvbiB0aGF0IGlzIHByaXZp bGVnZWQgYW5kL29yIGNvbmZpZGVudGlhbC4KIElmIHlvdSBhcmUgbm90IHRoZSBpbnRlbmRlZCBy ZWNpcGllbnQsIHlvdSBhcmUgaGVyZWJ5IG5vdGlmaWVkIHRoYXQKIGFueSByZXZpZXcsIHVzZSwg ZGlzc2VtaW5hdGlvbiwgZGlzY2xvc3VyZSBvciBjb3B5aW5nIG9mIHRoaXMgZW1haWwKIGFuZCBp dHMgYXR0YWNobWVudHMsIGlmIGFueSwgaXMgc3RyaWN0bHkgcHJvaGliaXRlZC4gIElmIHlvdSBo YXZlCiByZWNlaXZlZCB0aGlzIGVtYWlsIGluIGVycm9yLCBwbGVhc2UgaW1tZWRpYXRlbHkgbm90 aWZ5IHRoZSBzZW5kZXIgYnkKIHJldHVybiBlbWFpbCBhbmQgZGVsZXRlIHRoaXMgZW1haWwgZnJv bSB5b3VyIHN5c3RlbS4iCjwvZm9udD4KCg== From owner-freebsd-performance@FreeBSD.ORG Wed May 6 18:54:37 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40E4D10656ED for ; Wed, 6 May 2009 18:54:37 +0000 (UTC) (envelope-from Ggatten@waddell.com) Received: from mailhost0.waddell.com (mailhost0.waddell.com [12.154.38.61]) by mx1.freebsd.org (Postfix) with ESMTP id D9C1A8FC0A for ; Wed, 6 May 2009 18:54:36 +0000 (UTC) (envelope-from Ggatten@waddell.com) Received: from mailhost3.waddell.com (mailhost3.waddell.com [10.1.10.28]) by mailhost0.waddell.com (8.13.8/8.13.8) with ESMTP id n46I9kSn013174; Wed, 6 May 2009 13:09:56 -0500 (CDT) (envelope-from Ggatten@waddell.com) Received: from mailhost3.waddell.com (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id E9C1381450; Wed, 6 May 2009 13:09:45 -0500 (CDT) Received: from wadpexf0.waddell.com (wadpexf0.waddell.com [192.168.204.24]) by mailhost3.waddell.com (Postfix) with ESMTP id D2B698144E; Wed, 6 May 2009 13:09:45 -0500 (CDT) Received: from WADPEXV0.waddell.com ([192.168.204.25]) by wadpexf0.waddell.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 6 May 2009 13:09:42 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Wed, 6 May 2009 13:08:46 -0500 Message-ID: <70C0964126D66F458E688618E1CD008A0793EBD1@WADPEXV0.waddell.com> In-Reply-To: <4A01C202.8080803@seattlefenix.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: filesystem: 12h to delete 32GB of data Thread-Index: AcnObqfHTigM1AG/QCKN2todI36EswABn39A References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <4A01C202.8080803@seattlefenix.net> From: "Gary Gatten" To: "Benjamin Krueger" , "Wojciech Puchar" X-OriginalArrivalTime: 06 May 2009 18:09:42.0075 (UTC) FILETIME=[D40AE8B0:01C9CE75] X-Mailman-Approved-At: Wed, 06 May 2009 19:07:22 +0000 Cc: freebsd-performance@freebsd.org, Olivier Mueller , Bill Moran , freebsd-questions@freebsd.org Subject: RE: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 18:54:37 -0000 It could just be me, but I swear Hardware RAID has been faster for many many years, especially with RAID5 arrays - or anything that requires parity calcs. Most of my benchmarking was done on SCO OpenServer and Novell UnixWare and Netware, but hardware RAID controllers were always faster and of course required far less host CPU resources. Raid 0/1/10/0+1/whatever arrays, I recall weren't as drastic, but I can't imagine the controller making as big a difference as the drives in the array - unless of course the drive for said controller sux!
"This email is intended to be reviewed by only the intended recipient and may contain information that is privileged and/or confidential. If you are not the intended recipient, you are hereby notified that any review, use, dissemination, disclosure or copying of this email and its attachments, if any, is strictly prohibited. If you have received this email in error, please immediately notify the sender by return email and delete this email from your system." From owner-freebsd-performance@FreeBSD.ORG Wed May 6 19:21:56 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DD8E01065693; Wed, 6 May 2009 19:21:55 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (gate6.infracaninophile.co.uk [IPv6:2001:8b0:151:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 5D3188FC2A; Wed, 6 May 2009 19:21:55 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from happy-idiot-talk.infracaninophile.co.uk (localhost [IPv6:::1]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.3/8.14.3) with ESMTP id n46JLjhf002316; Wed, 6 May 2009 20:21:46 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: Sendmail DKIM Filter v2.8.2 smtp.infracaninophile.co.uk n46JLjhf002316 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=infracaninophile.co.uk; s=200708; t=1241637707; bh=Jl9iT116utVryz/yVoKH9Ycx4vi1/dRMC6BdjU1Bvn4=; h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References: In-Reply-To:Content-Type:Cc:Content-Type:Date:From:In-Reply-To: Message-ID:Mime-Version:References:To; z=Message-ID:=20<4A01E343.4020608@infracaninophile.co.uk>|Date:=20W ed,=2006=20May=202009=2020:21:39=20+0100|From:=20Matthew=20Seaman= 20|Organization:=20Infracaninophi le|User-Agent:=20Thunderbird=202.0.0.21=20(X11/20090420)|MIME-Vers ion:=201.0|To:=20Gary=20Gatten=20|CC:=20freeb sd-performance@freebsd.org,=20freebsd-questions@freebsd.org|Subjec t:=20Re:=20filesystem:=2012h=20to=20delete=2032GB=20of=20data|Refe rences:=20<70C0964126D66F458E688618E1CD008A0793EBD4@WADPEXV0.wadde ll.com>|In-Reply-To:=20<70C0964126D66F458E688618E1CD008A0793EBD4@W ADPEXV0.waddell.com>|X-Enigmail-Version:=200.95.6|Content-Type:=20 multipart/signed=3B=20micalg=3Dpgp-sha256=3B=0D=0A=20protocol=3D"a pplication/pgp-signature"=3B=0D=0A=20boundary=3D"------------enigD 7935878E099234D25665A93"; b=O/hp7luaPoCu2nJB2HIvnhkwnwWnldUymDndQQOw8eOMC3vBzY1aNGZl6ednKAS+e efA1tzCqc+2+Rqd90abJlam3946weQ0hzkxb4J/v6asi3N4bfq73iB3H5B9doDKMo7 0o2idp2vxpEmozGavC/krcdGE/3jdDQI8EtgkG9c= X-Authentication-Warning: happy-idiot-talk.infracaninophile.co.uk: Host localhost [IPv6:::1] claimed to be happy-idiot-talk.infracaninophile.co.uk Message-ID: <4A01E343.4020608@infracaninophile.co.uk> Date: Wed, 06 May 2009 20:21:39 +0100 From: Matthew Seaman Organization: Infracaninophile User-Agent: Thunderbird 2.0.0.21 (X11/20090420) MIME-Version: 1.0 To: Gary Gatten References: <70C0964126D66F458E688618E1CD008A0793EBD4@WADPEXV0.waddell.com> In-Reply-To: <70C0964126D66F458E688618E1CD008A0793EBD4@WADPEXV0.waddell.com> X-Enigmail-Version: 0.95.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigD7935878E099234D25665A93" X-Virus-Scanned: clamav-milter 0.95.1 at happy-idiot-talk.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-3.0 required=5.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VERIFIED,NO_RELAYS autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on happy-idiot-talk.infracaninophile.co.uk Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 19:21:58 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigD7935878E099234D25665A93 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Gary Gatten wrote: > OT now, but in high i/o envs with high concurrency needs, RAID5 is > still the way to go, esp if 90% of i/o is reads. Of course it depends > on file size / type as well... Anyway, let's sum it up with "a > storage subsystem is only as fast as its slowest link" It's not just the balance of reads over writes. It's the size and sequen= tial location of the IO requests. RAID5 is good for sequential reads -- eg. streaming a video -- where the system can read whole blocks from all the drives involved, calculate parity over the whole lot and then push all th= at blob of data up to the CPU. RAID5 is pretty pessimal if your usage pattern is small reads or writes randomly scattered over your storage area -- eg. typical RDBMS behaviour -- which works a great deal better on RAID10. I'd also contend that the essential difference between a really good fast= hardware raid controller and something disappointingly mundane is a decen= t amount of non-volatile cache memory. For most H/W raid that equates to using a battery backup unit. I've been thinking though that a few GB of fast solid-state hard drive configured as a gjournal for a RAID10 (ie gst= ripe +gmirror) might achieve the same effect for rather less outlay... It would probably not be too shabby with RAID5 even, but of course you'ld lose the benefit of offloading parity calculations onto the RAID controll= er's CPU. Still, modern multi-core CPUs are probably fast enough nowadays to make that viable for many purposes. =20 Cheers, Matthew --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW --------------enigD7935878E099234D25665A93 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.11 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEAREIAAYFAkoB40kACgkQ8Mjk52CukIxlQwCfZUK2JDHgQBeZ+hAkCZImW2pO SkEAoIyWUGFD7u0sDmqjueBr6w6TokAG =zm6f -----END PGP SIGNATURE----- --------------enigD7935878E099234D25665A93-- From owner-freebsd-performance@FreeBSD.ORG Wed May 6 20:51:09 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A378E1065688; Wed, 6 May 2009 20:51:09 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.29]) by mx1.freebsd.org (Postfix) with ESMTP id 4A3EA8FC0A; Wed, 6 May 2009 20:51:09 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so197724yxb.13 for ; Wed, 06 May 2009 13:51:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=/ea0+KbwWXYpS4GIsn6jOi79PDFqzj9iSMc256qlEMs=; b=mW5fJGiGyJ0OM7Q1bKrL7Fs0gndF4+Kk8DU8kgEdAomKE/q5GFiEiFLA5DL+thQ3tM CoLu1DmOzf4DJDqCcMJSp6INNdc7wxjBomRhzhX46cpFYikhq0xTEccxT/e6UX+I8c9Y eya+UijLW7UjIZqaxz3h2o0/kXQNMOiSqSNX4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=kXzadZ+i1ms53PZrF5hzp4vyQGgTbO2Nfj7+Ohr15zXMxeC0FXYxn5ZsV/a9KYR1MZ aNMPD02AxVNRDt+vb9R+3ubyQvVn2l5d1eiDe5fuf7WFgFFhcL2sy5YInQO8KAKWv0y5 JjWs69ClEPU8Rv8SchvWuPeK1g+p/BiZJBM4U= MIME-Version: 1.0 Received: by 10.150.201.17 with SMTP id y17mr3034097ybf.83.1241641836815; Wed, 06 May 2009 13:30:36 -0700 (PDT) In-Reply-To: <4A01E343.4020608@infracaninophile.co.uk> References: <70C0964126D66F458E688618E1CD008A0793EBD4@WADPEXV0.waddell.com> <4A01E343.4020608@infracaninophile.co.uk> Date: Wed, 6 May 2009 13:30:36 -0700 Message-ID: From: Freddie Cash To: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2009 20:51:10 -0000 On Wed, May 6, 2009 at 12:21 PM, Matthew Seaman wrote: > Gary Gatten wrote: >> OT now, but in high i/o envs with high concurrency needs, RAID5 is >> still the way to go, esp if 90% of i/o is reads. Of course it depends >> on file size / type as well... Anyway, let's sum it up with "a >> storage subsystem is only as fast as its slowest link" > > It's not just the balance of reads over writes. =C2=A0It's the size and > sequential location of the IO requests. =C2=A0RAID5 is good for sequentia= l reads -- eg. > streaming a video -- where the system can read whole blocks from all the > drives involved, calculate parity over the whole lot and then push all th= at > blob of data up to the CPU. > > RAID5 is pretty pessimal if your usage pattern is small reads or writes > randomly scattered over your storage area -- eg. typical RDBMS behaviour > -- which works a great deal better on RAID10. > > I'd also contend that the essential difference between a really good fast > hardware raid controller and something disappointingly mundane is a decen= t > amount of non-volatile cache memory. =C2=A0For most H/W raid that equates= to > using a battery backup unit. =C2=A0I've been thinking though that a few G= B of > fast solid-state hard drive configured as a gjournal for a RAID10 (ie > gstripe +gmirror) might achieve the same effect for rather less outlay...= =C2=A0It > would probably not be too shabby with RAID5 even, but of course you'ld > lose the benefit of offloading parity calculations onto the RAID > controller's CPU. Still, modern multi-core CPUs are probably fast enough = nowadays to > make that viable for many purposes. Depending on the number of drives you are using, ZFS would also be worth looking at. The raidz implementation works quite nicely, and (in theory) doesn't suffer from the major issues that RAID5/6 does. It also does implicit striping across all vdevs, so you can make some very fancy RAID layouts (each vdev can be mirrored, raidz1, raidz2, or just a bunch of disks). I don't know if the version of ZFS in FreeBSD 7.x supports hybrid pools, but the version in FreeBSD 8.0 should, which lets you add SSDs to the pool to be used automatically as "cache" in-between RAM and harddrives. --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-performance@FreeBSD.ORG Thu May 7 02:05:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5121106564A for ; Thu, 7 May 2009 02:05:18 +0000 (UTC) (envelope-from pathiaki2@yahoo.com) Received: from web110507.mail.gq1.yahoo.com (web110507.mail.gq1.yahoo.com [67.195.8.112]) by mx1.freebsd.org (Postfix) with SMTP id AE4898FC13 for ; Thu, 7 May 2009 02:05:18 +0000 (UTC) (envelope-from pathiaki2@yahoo.com) Received: (qmail 93550 invoked by uid 60001); 7 May 2009 01:38:38 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1241660318; bh=F366yw2xsX7PS4maDBOvBsRoihEie5FolNG+Af4DFac=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=sg8AQnx8gx+hp20d7WcB4goZyit36GyQKq2g62TVXPI2LfmuJ6pqiPotlOk7jvskA0V6m1XYZzchrMltyA9ZkH0+qMYkyhhmlF30pg3vMBOpOKQvuJkGDIuQU+x8uCfletgjzF6zuT+Zy+jw9+zFsaxn7EgABsM2Xpatc1Vm2f4= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=oq3xTwP6IyRH2sV1HGkNaY4x/18K3WzoiqhEIpGejCRhpwj+KfMuXIUQ+jz1a/fAG8zKHs14A0MLL6Xwrf7pS5yPPVP2XOSvF16fwUJuJqGyYwoywdTPbL0LggOnYSJ6sPObRulJU/vsntrlqtjrARl+MyH204bQJbDBfbLUicc=; Message-ID: <83156.91671.qm@web110507.mail.gq1.yahoo.com> X-YMail-OSG: WdccmvcVM1n45NiLIzsdb0xVIUAIS8abbZ7isDs_ltfMY1MD2GqGBuARjOJTsH6f1G6WYn5GbOXhhtLQTBpS18NgFnOYe0GZPaYPy9LEVXCrVJNc9UEuhfjVIZavVe0z_umjwvdoA8herSz.mNSAguO1On8RO4yxYEjcLec5TN.L5A24Gg1ePu13Aj2daJnRTMQslo9yYtmEMJ6LqRI_ykEmRAF0BZnJz5yTCSiufoGIhkX7kJzCPcjvIOR2w.8hMJ4vKSWgLjMPcAOGeEhxb4LTOAsFb8Mb_VDWj9DVdr1suLWfatExxpmj28y54VZueGak1oTuAHTugO.f5hQ4chxtZHoLbnY- Received: from [173.48.58.54] by web110507.mail.gq1.yahoo.com via HTTP; Wed, 06 May 2009 18:38:37 PDT X-Mailer: YahooMailRC/1277.35 YahooMailWebService/0.7.289.1 References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <4A01C202.8080803@seattlefenix.net> <70C0964126D66F458E688618E1CD008A0793EBD1@WADPEXV0.waddell.com> Date: Wed, 6 May 2009 18:38:37 -0700 (PDT) From: Paul Patterson To: Wojciech Puchar , Gary Gatten In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Benjamin Krueger , Olivier Mueller , freebsd-questions@freebsd.org, Bill Moran Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2009 02:05:19 -0000 Sorry. This statement is incorrect. If you aren't using ZFS, or even a GEOM volume with mirror/RAID5/softup/etc, you cannot make the statement that hardware RAID is faster. I learned that 3 years ago. It takes about 30 minutes to mirror 1.5TB on ZFS. Try that on hardware RAID. I did the same with 80 GB SATA drives a couple of years ago. Gmirror killed hardware mirror by 50% When your processor on your hardware RAID card is junk and you have a kickass processor and good chunk of memory on your main system and decent controller that isn't getting maxed, the "hardware RAID is always faster" paradigm walked out the door a few years ago. This does not go for EMC, IBM, Hitachi high-end storage arrays where you write to TBs of RAM Cache. P. ________________________________ From: Wojciech Puchar To: Gary Gatten Cc: freebsd-questions@freebsd.org; Benjamin Krueger ; Olivier Mueller ; freebsd-performance@freebsd.org; Bill Moran Sent: Wednesday, May 6, 2009 2:31:16 PM Subject: RE: filesystem: 12h to delete 32GB of data > It could just be me, but I swear Hardware RAID has been faster for many > many years, especially with RAID5 arrays - or anything that requires maybe with RAID5, but using RAID5 today (huge disk sizes, little sense to save on disk space) instead of RAID1/10 doesn't make much sense, as RAID5 is slow on writes by design _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Thu May 7 12:16:41 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1227106566B; Thu, 7 May 2009 12:16:41 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [IPv6:2001:4070:101:2::1]) by mx1.freebsd.org (Postfix) with ESMTP id 2B18D8FC1A; Thu, 7 May 2009 12:16:37 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (localhost [IPv6:::1]) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3) with ESMTP id n479fNK8039963; Thu, 7 May 2009 11:41:23 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.3/8.14.3/Submit) with ESMTP id n479fLpg039960; Thu, 7 May 2009 11:41:21 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Thu, 7 May 2009 11:41:21 +0200 (CEST) From: Wojciech Puchar To: Paul Patterson In-Reply-To: <83156.91671.qm@web110507.mail.gq1.yahoo.com> Message-ID: References: <1241610888.16418.64.camel@ompc.insign.local> <20090506084834.61600c42.wmoran@potentialtech.com> <4A01C202.8080803@seattlefenix.net> <70C0964126D66F458E688618E1CD008A0793EBD1@WADPEXV0.waddell.com> <83156.91671.qm@web110507.mail.gq1.yahoo.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Mailman-Approved-At: Thu, 07 May 2009 12:27:58 +0000 Cc: Gary Gatten , Olivier Mueller , Benjamin Krueger , Bill Moran , freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 May 2009 12:16:45 -0000 > If you aren't using ZFS, or even a GEOM volume with mirror/RAID5/softup/etc, > you cannot make the statement that hardware RAID is faster. I learned > that 3 years ago. i state exactly opposite. all hardware raid cards are made just to suck money from those who believe in it. like "performance is not enough - buy better/more expensive model." > This does not go for EMC, IBM, Hitachi high-end storage arrays where you write to TBs of RAM Cache. having same amount of extra memory on FreeBSD server directly will make better use of it. From owner-freebsd-performance@FreeBSD.ORG Fri May 8 11:43:45 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5230106566B for ; Fri, 8 May 2009 11:43:45 +0000 (UTC) (envelope-from seklecki@noc.cfi.pgh.pa.us) Received: from mx00.pub.collaborativefusion.com (drpmx.lab02.pitbpa0.pub.collaborativefusion.com [206.210.89.232]) by mx1.freebsd.org (Postfix) with ESMTP id 743C38FC19 for ; Fri, 8 May 2009 11:43:45 +0000 (UTC) (envelope-from seklecki@noc.cfi.pgh.pa.us) Received: from [192.168.2.161] (soundwave.ws.pitbpa0.priv.collaborativefusion.com [::ffff:192.168.2.161]) (AUTH: LOGIN seklecki, TLS: TLSv1/SSLv3,256bits,CAMELLIA256-SHA) by wingspan with esmtp; Fri, 08 May 2009 07:38:41 -0400 id 001BAC31.000000004A0419C1.00011549 From: "Brian A. Seklecki" To: Olivier Mueller In-Reply-To: <1241610888.16418.64.camel@ompc.insign.local> References: <1241610888.16418.64.camel@ompc.insign.local> Date: Fri, 08 May 2009 07:35:16 -0400 Message-Id: <1241782516.2053.10.camel@soundwave.ws.pitbpa0.priv.collaborativefusion.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 7bit X-Mailer: Evolution 2.24.5 (2.24.5-1.fc10) X-Mailman-Approved-At: Fri, 08 May 2009 11:47:41 +0000 Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: filesystem: 12h to delete 32GB of data X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 May 2009 11:43:46 -0000 On Wed, 2009-05-06 at 13:54 +0200, Olivier Mueller wrote: > -> it took about 12 hours to delete these 30GB of files and > sub-directories (smarty cache files: many small files in many dirs). Haven't you ever had the pleasure of running Sendmail on Solaris? :) Move this data store to a separate partition. When it comes time to burn the queue, stop the service, unmount the partition, newfs it, remount, restart svc. Long live Pisces v2. ~BAS