From owner-freebsd-performance@FreeBSD.ORG Thu Apr 8 15:43:12 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7FA1106564A for ; Thu, 8 Apr 2010 15:43:12 +0000 (UTC) (envelope-from ales@ripn.net) Received: from iruno.ripn.net (iruno.ripn.net [194.226.71.76]) by mx1.freebsd.org (Postfix) with ESMTP id 73A2F8FC16 for ; Thu, 8 Apr 2010 15:43:12 +0000 (UTC) Received: from [194.226.71.77] (helo=runo.ripn.net) by iruno.ripn.net with esmtp (RIPN) id 1Nztj0-000HRB-Ac for freebsd-performance@freebsd.org; Thu, 08 Apr 2010 19:32:58 +0400 Received: from ales by runo.ripn.net with esmtp (RIPN) id 1Nztj0-000HR8-5T for freebsd-performance@freebsd.org; Thu, 08 Apr 2010 19:32:58 +0400 From: "Alexandr D. Sergeev" To: freebsd-performance@freebsd.org Message-ID: <20100408153257.GA35557@runo.ripn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Virus-Scanned: ClamAV Date: Thu, 08 Apr 2010 19:32:58 +0400 Subject: Extreme performance falling on ZFS by durable operation of a server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Apr 2010 15:43:12 -0000 Hi, all! There are problems with a server on FreeBSD 7.2 CURRENT (6 Jul 2009), carrying out the task of mail server SMTP, IMAP. Messages are stored in maildirs on file system ZFS (mirror of 2 SAS 3k drives). Dovecot is used as IMAP server. After reboot the system works perfectly, however, somewhere in a month, at the same load, arise periodic long delays at opening of new letters. Judging by output iostat -x -w 1, during these moments the disk subsystem is strongly loaded. %busy disks comes nearer to 100 %. Readings nearby 300 per sec prevail. and approx. 10MBytes per sec. Somebody can advise, what diagnostics needed, for understanding of the reason of problems? There is an opinion that a problem with ZFS. Probably, it does not have enough memory. Because during the periods of occurrence of a problem, ARC is have less 300MB: vmstat -m ... solaris 1245732 274745K - 25872717949 16,32,64,128,256,512,1024,2048,4096 ... Or it is bug of ZFS implementation. -- Best regards, Alexandr D.Sergeev From owner-freebsd-performance@FreeBSD.ORG Thu Apr 8 20:44:13 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AEF6D1065677 for ; Thu, 8 Apr 2010 20:44:13 +0000 (UTC) (envelope-from kmatthew.macy@gmail.com) Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.27]) by mx1.freebsd.org (Postfix) with ESMTP id 64C7F8FC12 for ; Thu, 8 Apr 2010 20:44:13 +0000 (UTC) Received: by qw-out-2122.google.com with SMTP id 5so819941qwi.7 for ; Thu, 08 Apr 2010 13:44:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:reply-to:received :in-reply-to:references:date:x-google-sender-auth:received :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=MN6k6kzPwdY8JSjXURB344/1iLU+MDhaxrL9Rh3wf8Y=; b=JgoBfQ7ZFuNID9XAl7+jJedk+3qUbM6Ddp93We6s6e9gzf6OI8IPGmXm9Bz93rho9H Kx1Bo/I6Ou98pKT9sw4CJUugy190sCY9Uy1GtPnxxEstBTWmnn8TmIGyrzbJCP7BAMcm ZHR0XI0qGqHQ9gSvJn4mJKwKVIAoRyC3goKG4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:reply-to:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=eieVt/q8NPpzI7AHAVGkV9ghQXRANvqxhXIoKxpDvP+pCm7s9Qv31d6eGmVnD8uuTI A+z6S5hr5l/0u2aX7iYhmrs3ee9kYMs/dG+1OtM3C5LHtvCm+tK9z163/fO44GRAH+Eb 697UZ+GJ4kDSQSOSO6PIPz+X/cc/gkN1/i5t8= MIME-Version: 1.0 Sender: kmatthew.macy@gmail.com Received: by 10.229.214.136 with HTTP; Thu, 8 Apr 2010 13:14:42 -0700 (PDT) In-Reply-To: <20100408153257.GA35557@runo.ripn.net> References: <20100408153257.GA35557@runo.ripn.net> Date: Thu, 8 Apr 2010 13:14:42 -0700 X-Google-Sender-Auth: 5ed291a372e074b7 Received: by 10.229.212.213 with SMTP id gt21mr1007980qcb.2.1270757682200; Thu, 08 Apr 2010 13:14:42 -0700 (PDT) Message-ID: From: "K. Macy" To: "Alexandr D. Sergeev" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org Subject: Re: Extreme performance falling on ZFS by durable operation of a server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kmacy@freebsd.org List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Apr 2010 20:44:13 -0000 Hi Alexandr, - How much memory do you have? - What are your ARC settings? Thanks, Kip On Thu, Apr 8, 2010 at 8:32 AM, Alexandr D. Sergeev wrote: > Hi, all! > > There are problems with a server on FreeBSD 7.2 CURRENT (6 Jul 2009), car= rying out the task of mail server SMTP, IMAP. > > Messages are stored in maildirs on file system ZFS (mirror of 2 SAS 3k dr= ives). > Dovecot is used as IMAP server. > > After reboot the system works perfectly, however, somewhere in a month, a= t the same load, arise periodic long delays at opening of new letters. > > Judging by output iostat -x -w 1, during these moments the disk subsystem= is strongly loaded. > %busy disks comes nearer to 100 %. Readings nearby 300 per sec prevail. a= nd approx. 10MBytes per sec. > > Somebody can advise, what diagnostics needed, for understanding of the re= ason of problems? > > There is an opinion that a problem with ZFS. > > Probably, it does not have enough memory. Because during the periods of o= ccurrence of a problem, =A0ARC is have less 300MB: > > vmstat -m > ... > =A0 =A0 =A0solaris 1245732 274745K =A0 =A0 =A0 - 25872717949 =A016,32,64,= 128,256,512,1024,2048,4096 > ... > > Or it is bug of ZFS implementation. > > -- > Best regards, > Alexandr D.Sergeev > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= .org" > From owner-freebsd-performance@FreeBSD.ORG Fri Apr 9 02:44:13 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E1C0106566C for ; Fri, 9 Apr 2010 02:44:13 +0000 (UTC) (envelope-from ales@ripn.net) Received: from iruno.ripn.net (iruno.ripn.net [194.226.71.76]) by mx1.freebsd.org (Postfix) with ESMTP id 49E5B8FC14 for ; Fri, 9 Apr 2010 02:44:13 +0000 (UTC) Received: from [194.226.71.77] (helo=runo.ripn.net) by iruno.ripn.net with esmtp (RIPN) id 1O04CX-0001iq-GL; Fri, 09 Apr 2010 06:44:09 +0400 Received: from ales by runo.ripn.net with esmtp (RIPN) id 1O04CX-0001in-C8; Fri, 09 Apr 2010 06:44:09 +0400 From: "Alexandr D. Sergeev" To: "K. Macy" Message-ID: <20100409024409.GA1602@runo.ripn.net> References: <20100408153257.GA35557@runo.ripn.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Virus-Scanned: ClamAV Date: Fri, 09 Apr 2010 06:44:09 +0400 Cc: freebsd-performance@freebsd.org Subject: Re: Extreme performance falling on ZFS by durable operation of a server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Apr 2010 02:44:13 -0000 On Thu, Apr 08, 2010 at 01:14:42PM -0700, K. Macy wrote: > Hi Alexandr, Hi Kip, > > - How much memory do you have? 4Gb. Applications consume a lot of memory, is partially used swap (~1Gb, but without active i/o). We have planned increase in memory to 16G > > - What are your ARC settings? Settings is default, that recommended for amd64 kernel. But I am not assured that it is optimal. Thus sysctl variables are automatically sets to: vfs.zfs.arc_max ~800Mb vfs.zfs.arc_mim ~128Mb Now the next reboot was required. Together with which following settings have been made: vm.kmem_size=2048M vfs.zfs.arc_max=1536M vfs.zfs.arc_min=1024M vfs.zfs.vdev.cache.size=64M Let's look, as it will affect. However there is confidence that no problem at a size of ARC. Even at 200Mb ARC, productivity should not fall so strongly. > > Thanks, > Kip > > On Thu, Apr 8, 2010 at 8:32 AM, Alexandr D. Sergeev wrote: > > Hi, all! > > > > There are problems with a server on FreeBSD 7.2 CURRENT (6 Jul 2009), carrying out the task of mail server SMTP, IMAP. > > > > Messages are stored in maildirs on file system ZFS (mirror of 2 SAS 3k drives). > > Dovecot is used as IMAP server. > > > > After reboot the system works perfectly, however, somewhere in a month, at the same load, arise periodic long delays at opening of new letters. > > > > Judging by output iostat -x -w 1, during these moments the disk subsystem is strongly loaded. > > %busy disks comes nearer to 100 %. Readings nearby 300 per sec prevail. and approx. 10MBytes per sec. > > > > Somebody can advise, what diagnostics needed, for understanding of the reason of problems? > > > > There is an opinion that a problem with ZFS. > > > > Probably, it does not have enough memory. Because during the periods of occurrence of a problem,  ARC is have less 300MB: > > > > vmstat -m > > ... > >      solaris 1245732 274745K       - 25872717949  16,32,64,128,256,512,1024,2048,4096 > > ... > > > > Or it is bug of ZFS implementation. > > > > -- > > Best regards, > > Alexandr D.Sergeev > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > -- Best regards, Alexandr D.Sergeev