From owner-freebsd-performance@FreeBSD.ORG Mon Apr 12 09:11:42 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 557FA1065673 for ; Mon, 12 Apr 2010 09:11:42 +0000 (UTC) (envelope-from ales@ripn.net) Received: from iruno.ripn.net (iruno.ripn.net [194.226.71.76]) by mx1.freebsd.org (Postfix) with ESMTP id 107808FC17 for ; Mon, 12 Apr 2010 09:11:41 +0000 (UTC) Received: from [194.226.71.77] (helo=runo.ripn.net) by iruno.ripn.net with esmtp (RIPN) id 1O1Fg9-0006uO-7i; Mon, 12 Apr 2010 13:11:37 +0400 Received: from ales by runo.ripn.net with esmtp (RIPN) id 1O1Fg9-0006uL-46; Mon, 12 Apr 2010 13:11:37 +0400 From: "Alexandr D. Sergeev" To: freebsd-performance@freebsd.org Message-ID: <20100412091136.GE61157@runo.ripn.net> References: <20100408153257.GA35557@runo.ripn.net> <20100409024409.GA1602@runo.ripn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20100409024409.GA1602@runo.ripn.net> X-Virus-Scanned: ClamAV Date: Mon, 12 Apr 2010 13:11:37 +0400 Subject: Re: Extreme performance falling on ZFS by durable operation of a server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Apr 2010 09:11:42 -0000 Whether there are any ideas about the reasons of such problems? Whether transition to FreeBSD 8.0 RELEASE / STABLE can help? ZFS in it is declared as "bug free" On Fri, Apr 09, 2010 at 06:44:09AM +0400, Alexandr D. Sergeev wrote: > On Thu, Apr 08, 2010 at 01:14:42PM -0700, K. Macy wrote: > > Hi Alexandr, > > Hi Kip, > > > > > - How much memory do you have? > > 4Gb. > Applications consume a lot of memory, is partially used swap (~1Gb, but without active i/o). > > We have planned increase in memory to 16G > > > > > - What are your ARC settings? > > Settings is default, that recommended for amd64 kernel. > But I am not assured that it is optimal. > Thus sysctl variables are automatically sets to: > vfs.zfs.arc_max ~800Mb > vfs.zfs.arc_mim ~128Mb > > Now the next reboot was required. > Together with which following settings have been made: > > vm.kmem_size=2048M > vfs.zfs.arc_max=1536M > vfs.zfs.arc_min=1024M > vfs.zfs.vdev.cache.size=64M > > Let's look, as it will affect. > > However there is confidence that no problem at a size of ARC. > Even at 200Mb ARC, productivity should not fall so strongly. > > > > > Thanks, > > Kip > > > > On Thu, Apr 8, 2010 at 8:32 AM, Alexandr D. Sergeev wrote: > > > Hi, all! > > > > > > There are problems with a server on FreeBSD 7.2 CURRENT (6 Jul 2009), carrying out the task of mail server SMTP, IMAP. > > > > > > Messages are stored in maildirs on file system ZFS (mirror of 2 SAS 3k drives). > > > Dovecot is used as IMAP server. > > > > > > After reboot the system works perfectly, however, somewhere in a month, at the same load, arise periodic long delays at opening of new letters. > > > > > > Judging by output iostat -x -w 1, during these moments the disk subsystem is strongly loaded. > > > %busy disks comes nearer to 100 %. Readings nearby 300 per sec prevail. and approx. 10MBytes per sec. > > > > > > Somebody can advise, what diagnostics needed, for understanding of the reason of problems? > > > > > > There is an opinion that a problem with ZFS. > > > > > > Probably, it does not have enough memory. Because during the periods of occurrence of a problem,  ARC is have less 300MB: > > > > > > vmstat -m > > > ... > > >      solaris 1245732 274745K       - 25872717949  16,32,64,128,256,512,1024,2048,4096 > > > ... > > > > > > Or it is bug of ZFS implementation. > > > > > > -- > > > Best regards, > > > Alexandr D.Sergeev > > > _______________________________________________ > > > freebsd-performance@freebsd.org mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > > -- Best regards, Alexandr D.Sergeev From owner-freebsd-performance@FreeBSD.ORG Mon Apr 12 19:56:12 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 098321065673 for ; Mon, 12 Apr 2010 19:56:12 +0000 (UTC) (envelope-from kmatthew.macy@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.26]) by mx1.freebsd.org (Postfix) with ESMTP id AF4A88FC16 for ; Mon, 12 Apr 2010 19:56:10 +0000 (UTC) Received: by qw-out-2122.google.com with SMTP id 5so2086618qwi.7 for ; Mon, 12 Apr 2010 12:56:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:reply-to:received :in-reply-to:references:date:x-google-sender-auth:received :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=v6qzQscN2vfGCi6Lqzy0TtaBo3MwA75JgIapFD2Z944=; b=W2ZqznmxCsCMqjcT0FeJMbq5ty+7DWnaK5/ZAEi02z84aNqO/pMyJaMms3s3S3rG/q A+8XqHbVJKcKFDjhxOvEGD8FeASsEdby5vuHQqQ/+uDdzxiCQBEPCxO9fS8iI4niVl9m GIcl0JC0gR28h33FfgjsUTyWF7ARXTRg3iZak= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:reply-to:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=evs5LV85PZxBru6bSIswvGleAKXhRnGEH98dV+RtxhBR87OPygb+eWLrkEzR/WkcFN je+Bg6yjnYb1AuTQYDxXHhSx95VFO/Xg8cBlZLl7ZjnCeBcJbkcjKDgwGPnoK+H1WtKo 2+6vksyHJu7kAs/jfGIQmq+Mt9BNW9OF/9Tl8= MIME-Version: 1.0 Sender: kmatthew.macy@gmail.com Received: by 10.229.226.6 with HTTP; Mon, 12 Apr 2010 12:56:09 -0700 (PDT) In-Reply-To: <20100412091136.GE61157@runo.ripn.net> References: <20100408153257.GA35557@runo.ripn.net> <20100409024409.GA1602@runo.ripn.net> <20100412091136.GE61157@runo.ripn.net> Date: Mon, 12 Apr 2010 12:56:09 -0700 X-Google-Sender-Auth: 6e502a04439a2813 Received: by 10.229.242.85 with SMTP id lh21mr6432332qcb.67.1271102169847; Mon, 12 Apr 2010 12:56:09 -0700 (PDT) Message-ID: From: "K. Macy" To: "Alexandr D. Sergeev" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org Subject: Re: Extreme performance falling on ZFS by durable operation of a server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kmacy@freebsd.org List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Apr 2010 19:56:12 -0000 8-STABLE is definitely better, but you should also set arc_min to at least 384M-512M. I'm not sure that arc_min is even honored under 7.2. -Kip On Mon, Apr 12, 2010 at 2:11 AM, Alexandr D. Sergeev wrote: > Whether there are any ideas about the reasons of such problems? > > Whether transition to FreeBSD 8.0 RELEASE / STABLE can help? > ZFS in it is declared as "bug free" > > On Fri, Apr 09, 2010 at 06:44:09AM +0400, Alexandr D. Sergeev wrote: >> On Thu, Apr 08, 2010 at 01:14:42PM -0700, K. Macy wrote: >> > Hi Alexandr, >> >> Hi Kip, >> >> > >> > - How much memory do you have? >> >> 4Gb. >> Applications consume a lot of memory, is partially used swap (~1Gb, but = without active i/o). >> >> We have planned increase in memory to 16G >> >> > >> > - What are your ARC settings? >> >> Settings is default, that recommended for amd64 kernel. >> But I am not assured that it is optimal. >> Thus sysctl variables are automatically sets to: >> vfs.zfs.arc_max ~800Mb >> vfs.zfs.arc_mim ~128Mb >> >> Now the next reboot was required. >> Together with which following settings have been made: >> >> vm.kmem_size=3D2048M >> vfs.zfs.arc_max=3D1536M >> vfs.zfs.arc_min=3D1024M >> vfs.zfs.vdev.cache.size=3D64M >> >> Let's look, as it will affect. >> >> However there is confidence that no problem at a size of ARC. >> Even at 200Mb ARC, productivity should not fall so strongly. >> >> > >> > Thanks, >> > Kip >> > >> > On Thu, Apr 8, 2010 at 8:32 AM, Alexandr D. Sergeev wr= ote: >> > > Hi, all! >> > > >> > > There are problems with a server on FreeBSD 7.2 CURRENT (6 Jul 2009)= , carrying out the task of mail server SMTP, IMAP. >> > > >> > > Messages are stored in maildirs on file system ZFS (mirror of 2 SAS = 3k drives). >> > > Dovecot is used as IMAP server. >> > > >> > > After reboot the system works perfectly, however, somewhere in a mon= th, at the same load, arise periodic long delays at opening of new letters. >> > > >> > > Judging by output iostat -x -w 1, during these moments the disk subs= ystem is strongly loaded. >> > > %busy disks comes nearer to 100 %. Readings nearby 300 per sec preva= il. and approx. 10MBytes per sec. >> > > >> > > Somebody can advise, what diagnostics needed, for understanding of t= he reason of problems? >> > > >> > > There is an opinion that a problem with ZFS. >> > > >> > > Probably, it does not have enough memory. Because during the periods= of occurrence of a problem, =A0ARC is have less 300MB: >> > > >> > > vmstat -m >> > > ... >> > > =A0 =A0 =A0solaris 1245732 274745K =A0 =A0 =A0 - 25872717949 =A016,3= 2,64,128,256,512,1024,2048,4096 >> > > ... >> > > >> > > Or it is bug of ZFS implementation. >> > > >> > > -- >> > > Best regards, >> > > Alexandr D.Sergeev >> > > _______________________________________________ >> > > freebsd-performance@freebsd.org mailing list >> > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> > > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@fr= eebsd.org" >> > > >> > > -- > Best regards, > Alexandr D.Sergeev > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= .org" > From owner-freebsd-performance@FreeBSD.ORG Mon Apr 12 22:15:30 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45F85106564A; Mon, 12 Apr 2010 22:15:30 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from mail.digiware.nl (mail.ip6.digiware.nl [IPv6:2001:4cb8:1:106::2]) by mx1.freebsd.org (Postfix) with ESMTP id D1FD58FC08; Mon, 12 Apr 2010 22:15:29 +0000 (UTC) Received: from localhost (localhost.digiware.nl [127.0.0.1]) by mail.digiware.nl (Postfix) with ESMTP id 9C6BE15346D; Tue, 13 Apr 2010 00:15:27 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from mail.digiware.nl ([127.0.0.1]) by localhost (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1veTU93reVMQ; Tue, 13 Apr 2010 00:15:25 +0200 (CEST) Received: from [127.0.0.1] (unknown [192.168.10.212]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.digiware.nl (Postfix) with ESMTPSA id 0109015346C; Tue, 13 Apr 2010 00:15:24 +0200 (CEST) Message-ID: <4BC39B8C.6020808@digiware.nl> Date: Tue, 13 Apr 2010 00:15:40 +0200 From: Willem Jan Withagen User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: kmacy@freebsd.org References: <20100408153257.GA35557@runo.ripn.net> <20100409024409.GA1602@runo.ripn.net> <20100412091136.GE61157@runo.ripn.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 12 Apr 2010 22:23:26 +0000 Cc: freebsd-performance@freebsd.org, "Alexandr D. Sergeev" Subject: Re: Extreme performance falling on ZFS by durable operation of a server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Apr 2010 22:15:30 -0000 On 12-4-2010 21:56, K. Macy wrote: > 8-STABLE is definitely better, but you should also set arc_min to at > least 384M-512M. I'm not sure that arc_min is even honored under 7.2. Your mileage might vary.... I've been tempted to report similar problems running 8.0-stable march 23rd on amd64 with 8Gb running as fileserver (nfs/smb), imap/dovecot, rsync backup But I get stalls (of several seconds) when trying to do something in the shell first thing in the morning. After running some more service responce gets a bit more back to normal. Same holds for access over SMB and NFS with suffers from delays as well. Seen more of these type of reports with unconclusive answers thusfar, several of the suggestions did not improve the situation. Haven't found the time to very seriously trace this, and report more than just this. --WjW > On Mon, Apr 12, 2010 at 2:11 AM, Alexandr D. Sergeev wrote: >> Whether there are any ideas about the reasons of such problems? >> >> Whether transition to FreeBSD 8.0 RELEASE / STABLE can help? >> ZFS in it is declared as "bug free" >> >> On Fri, Apr 09, 2010 at 06:44:09AM +0400, Alexandr D. Sergeev wrote: >>> On Thu, Apr 08, 2010 at 01:14:42PM -0700, K. Macy wrote: >>>> Hi Alexandr, >>> >>> Hi Kip, >>> >>>> >>>> - How much memory do you have? >>> >>> 4Gb. >>> Applications consume a lot of memory, is partially used swap (~1Gb, but without active i/o). >>> >>> We have planned increase in memory to 16G >>> >>>> >>>> - What are your ARC settings? >>> >>> Settings is default, that recommended for amd64 kernel. >>> But I am not assured that it is optimal. >>> Thus sysctl variables are automatically sets to: >>> vfs.zfs.arc_max ~800Mb >>> vfs.zfs.arc_mim ~128Mb >>> >>> Now the next reboot was required. >>> Together with which following settings have been made: >>> >>> vm.kmem_size=2048M >>> vfs.zfs.arc_max=1536M >>> vfs.zfs.arc_min=1024M >>> vfs.zfs.vdev.cache.size=64M >>> >>> Let's look, as it will affect. >>> >>> However there is confidence that no problem at a size of ARC. >>> Even at 200Mb ARC, productivity should not fall so strongly. >>> >>>> >>>> Thanks, >>>> Kip >>>> >>>> On Thu, Apr 8, 2010 at 8:32 AM, Alexandr D. Sergeev wrote: >>>>> Hi, all! >>>>> >>>>> There are problems with a server on FreeBSD 7.2 CURRENT (6 Jul 2009), carrying out the task of mail server SMTP, IMAP. >>>>> >>>>> Messages are stored in maildirs on file system ZFS (mirror of 2 SAS 3k drives). >>>>> Dovecot is used as IMAP server. >>>>> >>>>> After reboot the system works perfectly, however, somewhere in a month, at the same load, arise periodic long delays at opening of new letters. >>>>> >>>>> Judging by output iostat -x -w 1, during these moments the disk subsystem is strongly loaded. >>>>> %busy disks comes nearer to 100 %. Readings nearby 300 per sec prevail. and approx. 10MBytes per sec. >>>>> >>>>> Somebody can advise, what diagnostics needed, for understanding of the reason of problems? >>>>> >>>>> There is an opinion that a problem with ZFS. >>>>> >>>>> Probably, it does not have enough memory. Because during the periods of occurrence of a problem, ARC is have less 300MB: >>>>> >>>>> vmstat -m >>>>> ... >>>>> solaris 1245732 274745K - 25872717949 16,32,64,128,256,512,1024,2048,4096 >>>>> ... >>>>> >>>>> Or it is bug of ZFS implementation. >>>>> >>>>> -- >>>>> Best regards, >>>>> Alexandr D.Sergeev >>>>> _______________________________________________ >>>>> freebsd-performance@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>>>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >>>>> >>> >> >> -- >> Best regards, >> Alexandr D.Sergeev >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Tue Apr 13 08:24:54 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 159F2106566C; Tue, 13 Apr 2010 08:24:54 +0000 (UTC) (envelope-from ales@iruno.ripn.net) Received: from iruno.ripn.net (iruno.ripn.net [194.226.71.76]) by mx1.freebsd.org (Postfix) with ESMTP id BAA028FC12; Tue, 13 Apr 2010 08:24:53 +0000 (UTC) Received: from ales by iruno.ripn.net with esmtp (RIPN) id 1O1bGf-000GFw-If; Tue, 13 Apr 2010 12:14:45 +0400 Message-ID: <4BC427F9.7090703@ripn.net> From: "Alexandr D. Sergeev" MIME-Version: 1.0 To: kmacy@freebsd.org References: <20100408153257.GA35557@runo.ripn.net> <20100409024409.GA1602@runo.ripn.net> <20100412091136.GE61157@runo.ripn.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: ClamAV Date: Tue, 13 Apr 2010 12:14:45 +0400 Cc: freebsd-performance@freebsd.org Subject: Re: Extreme performance falling on ZFS by durable operation of a server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Apr 2010 08:24:54 -0000 K. Macy пишет: > 8-STABLE is definitely better, but you should also set arc_min to at > least 384M-512M. I'm not sure that arc_min is even honored under 7.2. Thanks for advice! And I also not sure, because now settings is: vfs.zfs.arc_min: 1073741824 vfs.zfs.arc_max: 1610612736 but vmstat -m shows: solaris 1098941 622231K - 6443745881 16,32,64,128,256,512,1024,2048,4096 > > -Kip > > > On Mon, Apr 12, 2010 at 2:11 AM, Alexandr D. Sergeev wrote: >> Whether there are any ideas about the reasons of such problems? >> >> Whether transition to FreeBSD 8.0 RELEASE / STABLE can help? >> ZFS in it is declared as "bug free" >> >> On Fri, Apr 09, 2010 at 06:44:09AM +0400, Alexandr D. Sergeev wrote: >>> On Thu, Apr 08, 2010 at 01:14:42PM -0700, K. Macy wrote: >>>> Hi Alexandr, >>> Hi Kip, >>> >>>> - How much memory do you have? >>> 4Gb. >>> Applications consume a lot of memory, is partially used swap (~1Gb, but without active i/o). >>> >>> We have planned increase in memory to 16G >>> >>>> - What are your ARC settings? >>> Settings is default, that recommended for amd64 kernel. >>> But I am not assured that it is optimal. >>> Thus sysctl variables are automatically sets to: >>> vfs.zfs.arc_max ~800Mb >>> vfs.zfs.arc_mim ~128Mb >>> >>> Now the next reboot was required. >>> Together with which following settings have been made: >>> >>> vm.kmem_size=2048M >>> vfs.zfs.arc_max=1536M >>> vfs.zfs.arc_min=1024M >>> vfs.zfs.vdev.cache.size=64M >>> >>> Let's look, as it will affect. >>> >>> However there is confidence that no problem at a size of ARC. >>> Even at 200Mb ARC, productivity should not fall so strongly. >>> >>>> Thanks, >>>> Kip >>>> >>>> On Thu, Apr 8, 2010 at 8:32 AM, Alexandr D. Sergeev wrote: >>>>> Hi, all! >>>>> >>>>> There are problems with a server on FreeBSD 7.2 CURRENT (6 Jul 2009), carrying out the task of mail server SMTP, IMAP. >>>>> >>>>> Messages are stored in maildirs on file system ZFS (mirror of 2 SAS 3k drives). >>>>> Dovecot is used as IMAP server. >>>>> >>>>> After reboot the system works perfectly, however, somewhere in a month, at the same load, arise periodic long delays at opening of new letters. >>>>> >>>>> Judging by output iostat -x -w 1, during these moments the disk subsystem is strongly loaded. >>>>> %busy disks comes nearer to 100 %. Readings nearby 300 per sec prevail. and approx. 10MBytes per sec. >>>>> >>>>> Somebody can advise, what diagnostics needed, for understanding of the reason of problems? >>>>> >>>>> There is an opinion that a problem with ZFS. >>>>> >>>>> Probably, it does not have enough memory. Because during the periods of occurrence of a problem, ARC is have less 300MB: >>>>> >>>>> vmstat -m >>>>> ... >>>>> solaris 1245732 274745K - 25872717949 16,32,64,128,256,512,1024,2048,4096 >>>>> ... >>>>> >>>>> Or it is bug of ZFS implementation. >>>>> >>>>> -- >>>>> Best regards, >>>>> Alexandr D.Sergeev >>>>> _______________________________________________ >>>>> freebsd-performance@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >>>>> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >>>>> >> -- >> Best regards, >> Alexandr D.Sergeev >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >> -- Best regards, Alexandr D.Sergeev From owner-freebsd-performance@FreeBSD.ORG Tue Apr 13 20:57:54 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B62B1065675 for ; Tue, 13 Apr 2010 20:57:54 +0000 (UTC) (envelope-from cwf-ml@arcor.de) Received: from mail.arcor.net (mail.arcor.net [145.253.32.2]) by mx1.freebsd.org (Postfix) with ESMTP id AC62B8FC14 for ; Tue, 13 Apr 2010 20:57:53 +0000 (UTC) Received: from mail-gw.arcor.net (sunmaildns02.arcor.net [62.213.136.74]) by mail.arcor.net (8.13.8/8.13.8) with ESMTP id o3DKUZCG010458; Tue, 13 Apr 2010 22:30:35 +0200 (MEST) Received: from tkmail.arcor.net (esbanl-1ug-1-p2.arcor.net [62.213.137.69]) by mail-gw.arcor.net (8.13.8/8.13.8) with ESMTP id o3DKUZVj008008; Tue, 13 Apr 2010 22:30:35 +0200 (MEST) Received: from sky.tnd.arcor.net (sky.tnd.arcor.net [10.128.188.151]) by tkmail.arcor.net (Postfix) with ESMTP id B279B1F9E; Tue, 13 Apr 2010 22:30:35 +0200 (MEST) Received: from [10.128.189.1] (wefapc.tnd.arcor.net [10.128.189.1]) by sky.tnd.arcor.net (Postfix) with ESMTP id 966DB2F12; Tue, 13 Apr 2010 22:30:35 +0200 (MEST) Message-ID: <4BC4D46B.9040805@arcor.de> Date: Tue, 13 Apr 2010 22:30:35 +0200 From: Christoph Weber-Fahr User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: freebsd-proliant@freebsd.org, freebsd-performance@freebsd.org X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Subject: is trhere a problem with the HP SmartArray P410 and/or Postgres on FreeBSD ? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Apr 2010 20:57:54 -0000 Hello, on a new HP Proliant DL385 G6 I have a P410 with BBWC and 8 hard drives in RAID5. (BBWC is Battery Backed Write Cache Enabler, and the controller is configured with 300M (75%) write cache). One of the applications we want to run is PostgreSQL (not the top priority, else we would use a different RAID scenario). But before getting the system live, we tested the performance. PostgreSQL gets a whole wopping 80 (!) tps out of this scenario wit pgbench ( -c 5 -t 5000) Now I had not expected top performance, but something to the tune of 500 tps should be well possible. And it's definitely a storage or filke system issue, since CPU is bored to death during the benchmark, while the file system is at 100% according to systat. On a comparable Hardware our test machine runs as a VM under VMWare and gets ~600 tps. Is there a known problem with this setup? OS is FreeBSD-amd64 7.2-p7 . Regards Christoph Wenber-Fahr From owner-freebsd-performance@FreeBSD.ORG Wed Apr 14 09:04:40 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22D72106566B for ; Wed, 14 Apr 2010 09:04:40 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id CE5848FC0A for ; Wed, 14 Apr 2010 09:04:39 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1O1yWU-00070G-Fl for freebsd-performance@freebsd.org; Wed, 14 Apr 2010 11:04:38 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 Apr 2010 11:04:38 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 Apr 2010 11:04:38 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 14 Apr 2010 11:04:41 +0200 Lines: 39 Message-ID: References: <4BC4D46B.9040805@arcor.de> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.8) Gecko/20100329 Thunderbird/3.0.3 In-Reply-To: <4BC4D46B.9040805@arcor.de> X-Enigmail-Version: 1.0.1 Subject: Re: is trhere a problem with the HP SmartArray P410 and/or Postgres on FreeBSD ? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Apr 2010 09:04:40 -0000 On 04/13/10 22:30, Christoph Weber-Fahr wrote: > Hello, > > on a new HP Proliant DL385 G6 I have a P410 with BBWC and > 8 hard drives in RAID5. > (BBWC is Battery Backed Write Cache Enabler, and the controller > is configured with 300M (75%) write cache). > > One of the applications we want to run is PostgreSQL (not the top > priority, else we would use a different RAID scenario). > > But before getting the system live, we tested the performance. > PostgreSQL gets a whole wopping 80 (!) tps out of this > scenario wit pgbench ( -c 5 -t 5000) Hmm yes, this is too slow. > Now I had not expected top performance, but something to the tune > of 500 tps should be well possible. And it's definitely a storage Yes, your expectations are correct. > or filke system issue, since CPU is bored to death during the > benchmark, while the file system is at 100% according to systat. > On a comparable Hardware our test machine runs as a VM under VMWare > and gets ~600 tps. > > Is there a known problem with this setup? > > OS is FreeBSD-amd64 7.2-p7 . Can you give more details about your hardware? CPU? Memory? File system? The "100% sys time" point makes it possible it's not the storage itself that is the issue. You can also collect more information: I'd recommend with either "iostat -x 1" or "gstat". From owner-freebsd-performance@FreeBSD.ORG Wed Apr 14 13:54:06 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B410F106564A for ; Wed, 14 Apr 2010 13:54:06 +0000 (UTC) (envelope-from cwf-ml@arcor.de) Received: from mail.arcor.net (mail.arcor.net [145.253.32.2]) by mx1.freebsd.org (Postfix) with ESMTP id 3FF618FC0C for ; Wed, 14 Apr 2010 13:54:05 +0000 (UTC) Received: from mail-gw.arcor.net (sunmaildns01.arcor.net [62.213.136.73]) by mail.arcor.net (8.13.8/8.13.8) with ESMTP id o3EDs4LB025306 for ; Wed, 14 Apr 2010 15:54:04 +0200 (MEST) Received: from tkmail.arcor.net (esbanl-1ug-1-p2.arcor.net [62.213.137.69]) by mail-gw.arcor.net (8.13.8/8.13.8) with ESMTP id o3EDs4Vf023629 for ; Wed, 14 Apr 2010 15:54:04 +0200 (MEST) Received: from sky.tnd.arcor.net (sky.tnd.arcor.net [10.128.188.151]) by tkmail.arcor.net (Postfix) with ESMTP id BDF3B1FFF for ; Wed, 14 Apr 2010 15:54:03 +0200 (MEST) Received: from [10.128.189.1] (wefapc.tnd.arcor.net [10.128.189.1]) by sky.tnd.arcor.net (Postfix) with ESMTP id 9C1942F4A for ; Wed, 14 Apr 2010 15:54:03 +0200 (MEST) Message-ID: <4BC5C8FA.3030505@arcor.de> Date: Wed, 14 Apr 2010 15:54:02 +0200 From: Christoph Weber-Fahr User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4BC4D46B.9040805@arcor.de> In-Reply-To: X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: is there a problem with the HP SmartArray P410 and/or Postgres on FreeBSD ? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Apr 2010 13:54:06 -0000 Hello, On 14.04.2010 11:04, Ivan Voras wrote: > On 04/13/10 22:30, Christoph Weber-Fahr wrote: >> Hello, >> >> on a new HP Proliant DL385 G6 I have a P410 with BBWC and >> 8 hard drives in RAID5. >> (BBWC is Battery Backed Write Cache Enabler, and the controller >> is configured with 300M (75%) write cache). >> >> One of the applications we want to run is PostgreSQL (not the top >> priority, else we would use a different RAID scenario). >> >> But before getting the system live, we tested the performance. >> PostgreSQL gets a whole wopping 80 (!) tps out of this >> scenario wit pgbench ( -c 5 -t 5000) > Can you give more details about your hardware? CPU? Memory? File system? > The "100% sys time" point makes it possible it's not the storage itself > that is the issue. CPU is a single AMD Athlon 2427 6-core processor, with 4 Gig RAM. Filesystem is standard bsd ufs/ffs with softupdates on a 3.2 TB RAID5. As for 100%sys time, I didn't claim that. I just note the storage device da0 in the systat -vmstat screen is marked as 100% busy. > You can also collect more information: I'd recommend with either "iostat > -x 1" or "gstat". I'll append some stats. Regards Christoph Weber-Fahr iostat -x 1 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 1.1 16.4 15.4 1030.9 1 5.5 7 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 57.0 0.0 3645.3 1 18.1 100 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 400.5 0.0 24052.3 1 2.6 96 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 196.8 0.0 12593.5 1 5.2 97 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 61.9 0.0 3963.4 1 16.5 98 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 70.9 0.0 4538.8 1 14.5 99 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 61.9 0.0 3963.4 1 16.6 100 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 60.9 0.0 3899.5 1 16.5 97 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 76.9 0.0 4922.3 1 13.9 101 pass0 0.0 0.0 0.0 0.0 0 0.0 0 extended device statistics device r/s w/s kr/s kw/s wait svc_t %b da0 0.0 62.9 0.0 4027.3 1 16.4 100 pass0 0.0 0.0 0.0 0.0 0 0.0 0 gstat dT: 1.001s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 0 0 0 0.0 0 0 0.0 0.0| acd0 1 68 0 0 0.0 68 4347 14.8 99.4| da0 0 0 0 0 0.0 0 0 0.0 0.0| da0p1 0 0 0 0 0.0 0 0 0.0 0.0| da0p2 0 0 0 0 0.0 0 0 0.0 0.0| da0p3 0 0 0 0 0.0 0 0 0.0 0.0| da0p4 0 0 0 0 0.0 0 0 0.0 0.0| da0p5 1 68 0 0 0.0 68 4347 14.9 99.6| da0p6 From owner-freebsd-performance@FreeBSD.ORG Wed Apr 14 14:10:53 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E52E3106564A for ; Wed, 14 Apr 2010 14:10:53 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 6F84D8FC14 for ; Wed, 14 Apr 2010 14:10:53 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1O23Iq-0004oH-BV for freebsd-performance@freebsd.org; Wed, 14 Apr 2010 16:10:52 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 Apr 2010 16:10:52 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 14 Apr 2010 16:10:52 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 14 Apr 2010 16:10:54 +0200 Lines: 105 Message-ID: References: <4BC4D46B.9040805@arcor.de> <4BC5C8FA.3030505@arcor.de> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.8) Gecko/20100329 Thunderbird/3.0.3 In-Reply-To: <4BC5C8FA.3030505@arcor.de> X-Enigmail-Version: 1.0.1 Subject: Re: is there a problem with the HP SmartArray P410 and/or Postgres on FreeBSD ? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Apr 2010 14:10:54 -0000 On 04/14/10 15:54, Christoph Weber-Fahr wrote: > Hello, > > On 14.04.2010 11:04, Ivan Voras wrote: >> On 04/13/10 22:30, Christoph Weber-Fahr wrote: >>> Hello, >>> >>> on a new HP Proliant DL385 G6 I have a P410 with BBWC and >>> 8 hard drives in RAID5. >>> (BBWC is Battery Backed Write Cache Enabler, and the controller >>> is configured with 300M (75%) write cache). >>> >>> One of the applications we want to run is PostgreSQL (not the top >>> priority, else we would use a different RAID scenario). >>> >>> But before getting the system live, we tested the performance. >>> PostgreSQL gets a whole wopping 80 (!) tps out of this >>> scenario wit pgbench ( -c 5 -t 5000) > >> Can you give more details about your hardware? CPU? Memory? File system? >> The "100% sys time" point makes it possible it's not the storage itself >> that is the issue. > > CPU is a single AMD Athlon 2427 6-core processor, with 4 Gig RAM. > > Filesystem is standard bsd ufs/ffs with softupdates on a 3.2 TB RAID5. > > As for 100%sys time, I didn't claim that. I just note the storage > device da0 in the systat -vmstat screen is marked as 100% busy. Hi, Ok, thanks for clarifying that. In this case, and looking at the stats below, I think you are right - something really is problematic with your storage. Unfortunately, I don't clearly see what could be the issue, except if your BBU is not enabled or the controller isn't set for write-through. I have a P400 on a machine and it works as expected (on 6 drives in RAID-10). What are your sequential read and write rates? Try testing with bonnie++. I get > 200 MB/s in either direction but rewrite rates of 38 MB/s. It also reports seek rate of ~~850/s. > iostat -x 1 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 1.1 16.4 15.4 1030.9 1 5.5 7 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 57.0 0.0 3645.3 1 18.1 100 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 400.5 0.0 24052.3 1 2.6 96 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 196.8 0.0 12593.5 1 5.2 97 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 61.9 0.0 3963.4 1 16.5 98 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 70.9 0.0 4538.8 1 14.5 99 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 61.9 0.0 3963.4 1 16.6 100 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 60.9 0.0 3899.5 1 16.5 97 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 76.9 0.0 4922.3 1 13.9 101 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > extended device statistics > device r/s w/s kr/s kw/s wait svc_t %b > da0 0.0 62.9 0.0 4027.3 1 16.4 100 > pass0 0.0 0.0 0.0 0.0 0 0.0 0 > > gstat > > dT: 1.001s w: 1.000s > L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name > 0 0 0 0 0.0 0 0 0.0 0.0| acd0 > 1 68 0 0 0.0 68 4347 14.8 99.4| da0 > 0 0 0 0 0.0 0 0 0.0 0.0| da0p1 > 0 0 0 0 0.0 0 0 0.0 0.0| da0p2 > 0 0 0 0 0.0 0 0 0.0 0.0| da0p3 > 0 0 0 0 0.0 0 0 0.0 0.0| da0p4 > 0 0 0 0 0.0 0 0 0.0 0.0| da0p5 > 1 68 0 0 0.0 68 4347 14.9 99.6| da0p6 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >