From owner-freebsd-performance@FreeBSD.ORG Sat Jul 24 23:06:55 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D33081065672; Sat, 24 Jul 2010 23:06:55 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 8F4118FC12; Sat, 24 Jul 2010 23:06:55 +0000 (UTC) Received: by fxm13 with SMTP id 13so6110651fxm.13 for ; Sat, 24 Jul 2010 16:06:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=oF/QTcyggo/Ns82j6M4wI8yk7etZ8ON5En1aM9oUpkA=; b=JgX2InMMxUmCy50M0li4wSVWBFOQgfTQRqFmBemIdg/ERUTmQyLJmdZPeBEyk9IJwC 785LsPbFk3AK9DV4krjYI/bP8a8Omj2VaXFDW1lLj2Elmge3R786cE49DvBlVVLIdMCa pken2i8RN6R1hV8XtqpJDfk/B8TXw+hD5Mo8c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=SBR0kxHnKeo52rH8kd1eLoMUgmz2/msIEtvxj5e8xMUHB7Ha1+rqfa7wIKhg1I7szy eS/4/Lv0t6GqC7WBqrm/7dfzwuHP0l6mAFCSzmBfET6DjnPiciB/8SKZ9b9wVTnUBhXz AEN707NP/c98DXsjNxjAglJ2aoEPTCBL8BoTo= Received: by 10.223.126.68 with SMTP id b4mr4713168fas.96.1280012814031; Sat, 24 Jul 2010 16:06:54 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (pc.mavhome.dp.ua [212.86.226.226]) by mx.google.com with ESMTPS id r8sm690903faq.10.2010.07.24.16.06.52 (version=SSLv3 cipher=RC4-MD5); Sat, 24 Jul 2010 16:06:53 -0700 (PDT) Sender: Alexander Motin Message-ID: <4C4B720A.6020802@FreeBSD.org> Date: Sun, 25 Jul 2010 02:06:50 +0300 From: Alexander Motin User-Agent: Thunderbird 2.0.0.24 (X11/20100402) MIME-Version: 1.0 To: Rui Paulo References: <4C4AF046.40507@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sun, 25 Jul 2010 00:36:11 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Intel TurboBoost in practice X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jul 2010 23:06:56 -0000 Rui Paulo wrote: > On 24 Jul 2010, at 14:53, Alexander Motin wrote: >> Here is my test case: FreeBSD 9-CURRENT on Core i5 650 CPU, 3.2GHz + 1/2 >> TurboBoost steps (+133/+266MHz) with boxed cooler at the open air. I was >> measuring building time of the net/mpd5 from sources, using only one CPU >> core (cpuset -l 0 time make). >> >> Untuned system (hz=1000): 14.15 sec >> Enabled ACPI C2 (hz=1000+C2): 13.85 sec >> Enabled ACPI C3 (hz=1000+C3): 13.91 sec >> Reduced HZ (hz=100): 14.16 sec >> Enabled ACPI C2 (hz=100+C2): 13.85 sec >> Enabled ACPI C3 (hz=100+C3): 13.86 sec >> Timers tuned* (hz=100): 14.10 sec >> Enabled ACPI C2 (hz=100+C2): 13.71 sec >> Enabled ACPI C3 (hz=100+C3): 13.73 sec >> >> All numbers tested few times and are repeatable up to +/-0.01sec. >> >> PS: In this case benefit is small, but it is the least that can be >> achieved, depending on CPU model. Some models allow frequency to be >> risen by up to 6 steps (+798MHz). > > The numbers that you are showing doesn't show much difference. Have you tried buildworld? If you mean relative difference -- as I have told, it's mostly because of my CPU. It's maximal boost is 266MHz (8.3%), but 133MHz of them is enabled most of time if CPU is not overheated. It probably doesn't, as it works on clear table under air conditioner. So maximal effect I can expect on is 4.2%. In such situation 2.8% probably not so bad to illustrate that feature works and there is space for further improvements. If I had Core i5-750S I would expect 33% boost. If you mean absolute difference, here are results or four buildworld runs: hw.acpi.cpu.cx_lowest=C1: 4654.23 sec hw.acpi.cpu.cx_lowest=C2: 4556.37 sec hw.acpi.cpu.cx_lowest=C2: 4570.85 sec hw.acpi.cpu.cx_lowest=C1: 4679.83 sec Benefit is about 2.1%. Each time results were erased and sources pre-cached into RAM. Storage was SSD, so disk should not be an issue. -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Mon Jul 26 12:19:37 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7088B1065675; Mon, 26 Jul 2010 12:19:37 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 4FBFD8FC0C; Mon, 26 Jul 2010 12:19:37 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id CADC946B3B; Mon, 26 Jul 2010 08:19:36 -0400 (EDT) Date: Mon, 26 Jul 2010 13:19:36 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Alexander Motin In-Reply-To: <4C4B720A.6020802@FreeBSD.org> Message-ID: References: <4C4AF046.40507@FreeBSD.org> <4C4B720A.6020802@FreeBSD.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Rui Paulo Subject: Re: Intel TurboBoost in practice X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jul 2010 12:19:37 -0000 On Sun, 25 Jul 2010, Alexander Motin wrote: >> The numbers that you are showing doesn't show much difference. Have you >> tried buildworld? > > If you mean relative difference -- as I have told, it's mostly because of my > CPU. It's maximal boost is 266MHz (8.3%), but 133MHz of them is enabled most > of time if CPU is not overheated. It probably doesn't, as it works on clear > table under air conditioner. So maximal effect I can expect on is 4.2%. In > such situation 2.8% probably not so bad to illustrate that feature works and > there is space for further improvements. If I had Core i5-750S I would > expect 33% boost. Can I recommend the use of ministat(1) and sample sizes of at least 8 runs per configuration? Robert > > If you mean absolute difference, here are results or four buildworld runs: > hw.acpi.cpu.cx_lowest=C1: 4654.23 sec > hw.acpi.cpu.cx_lowest=C2: 4556.37 sec > hw.acpi.cpu.cx_lowest=C2: 4570.85 sec > hw.acpi.cpu.cx_lowest=C1: 4679.83 sec > Benefit is about 2.1%. Each time results were erased and sources > pre-cached into RAM. Storage was SSD, so disk should not be an issue. > > -- > Alexander Motin > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Mon Jul 26 14:12:26 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB2C11065673; Mon, 26 Jul 2010 14:12:26 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1BD528FC12; Mon, 26 Jul 2010 14:12:25 +0000 (UTC) Received: by fxm13 with SMTP id 13so119931fxm.13 for ; Mon, 26 Jul 2010 07:12:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=G7oTVotFxTUAVpJGOpyCve/ZLseePrYacSwaTC6aNno=; b=QbPZspZFmIbc++j9Ib4gbVezU1T2mNbNj99c/AOMgOMOH0K5BjnWEnBYVBPJ7VxOJ+ wLtBsL+xUntTjL0rTnDQKh3qRjIDkwGLs8MDru4PAGmPBrPm6K15ejm5lT1IMPkKnVAs K/mgn4DrCMuHg4SGSiLX8s0hn/V1ZqfEyUGhw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=Gzu6YNHAKGwISQKktLl7d04TmFpIszqfBmLdg+Y296sFpSZ0tbiMk6RHP+ZymPEhJL Jmrs/uPjH364K4syI3ULulkkGR57spgefllJNdIzPZaWDFZ0jJHd09UCEANPl3YLNmK/ PdhOu3tRT3+711/SFFJhM0HlVxJrwam4Jgc7o= Received: by 10.223.109.140 with SMTP id j12mr6505414fap.22.1280153544092; Mon, 26 Jul 2010 07:12:24 -0700 (PDT) Received: from mavbook2.mavhome.dp.ua (pc.mavhome.dp.ua [212.86.226.226]) by mx.google.com with ESMTPS id w11sm1401657fao.13.2010.07.26.07.12.21 (version=SSLv3 cipher=RC4-MD5); Mon, 26 Jul 2010 07:12:22 -0700 (PDT) Sender: Alexander Motin Message-ID: <4C4D9779.8080505@FreeBSD.org> Date: Mon, 26 Jul 2010 17:11:05 +0300 From: Alexander Motin User-Agent: Thunderbird 2.0.0.23 (X11/20091212) MIME-Version: 1.0 To: Robert Watson References: <4C4AF046.40507@FreeBSD.org> <4C4B720A.6020802@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 26 Jul 2010 15:56:02 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Rui Paulo Subject: Re: Intel TurboBoost in practice X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jul 2010 14:12:26 -0000 Robert Watson wrote: > On Sun, 25 Jul 2010, Alexander Motin wrote: >>> The numbers that you are showing doesn't show much difference. Have >>> you tried buildworld? >> >> If you mean relative difference -- as I have told, it's mostly because >> of my CPU. It's maximal boost is 266MHz (8.3%), but 133MHz of them is >> enabled most of time if CPU is not overheated. It probably doesn't, as >> it works on clear table under air conditioner. So maximal effect I can >> expect on is 4.2%. In such situation 2.8% probably not so bad to >> illustrate that feature works and there is space for further >> improvements. If I had Core i5-750S I would expect 33% boost. > > Can I recommend the use of ministat(1) and sample sizes of at least 8 > runs per configuration? Thanks for pushing me to do it right. :) Here is 3*15 runs with fresh kernel with disabled debug. Results are quite close to original: -2.73% and -2.19% of time. x C1 + C2 * C3 +-----------------------------------------------------------------+ |+ * x | |+ * x | |+ * x | |+ * x | |+ * x | |+ * x | |+ * x | |+ ** x | |+ + ** xx | |+ + ** ** xx x| | |__M_A____| | |A| | | |A| | +-----------------------------------------------------------------+ N Min Max Median Avg Stddev x 15 12.68 12.84 12.69 12.698667 0.039254966 + 15 12.35 12.36 12.35 12.351333 0.0035186578 Difference at 95.0% confidence -0.347333 +/- 0.0208409 -2.7352% +/- 0.164119% (Student's t, pooled s = 0.0278687) * 15 12.41 12.44 12.42 12.42 0.0075592895 Difference at 95.0% confidence -0.278667 +/- 0.0211391 -2.19446% +/- 0.166467% (Student's t, pooled s = 0.0282674) I also checked one more aspect -- TurboBoost works only when CPU runs at highest EIST frequency (P0 state). I've reduced dev.cpu.0.freq from 3201 to 3067 and repeated the test: x C1 + C2 * C3 +-----------------------------------------------------------------+ | x + * | | x + * | | x + * | | x + * *| | x x + * *| | x x + + * *| | x x + + * *| | x x + + * *| | x x + + + + * *| ||MA| | | |_MA_| | | M_A_|| +-----------------------------------------------------------------+ N Min Max Median Avg Stddev x 15 13.72 13.73 13.72 13.723333 0.0048795004 + 15 13.79 13.82 13.8 13.803333 0.0072374686 Difference at 95.0% confidence 0.08 +/- 0.00461567 0.582949% +/- 0.0336337% (Student's t, pooled s = 0.00617213) * 15 13.89 13.9 13.89 13.894 0.0050709255 Difference at 95.0% confidence 0.170667 +/- 0.00372127 1.24362% +/- 0.0271164% (Student's t, pooled s = 0.00497613) In that case using C2 or C3 predictably caused small performance reduce, as after falling to sleep, CPU needs time to wakeup. Even if tested CPU0 won't ever sleep during test, it's TLB shutdown IPIs to other cores still probably could suffer from waiting other cores' wakeup. Obviously in first test these 0.58% and 1.24% were subtracted from the TurboBoost's maximal benefit of 4.3% on this CPU. -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Tue Jul 27 17:17:47 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D97C1065677; Tue, 27 Jul 2010 17:17:47 +0000 (UTC) (envelope-from alan.l.cox@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 38DDB8FC16; Tue, 27 Jul 2010 17:17:46 +0000 (UTC) Received: by pwj9 with SMTP id 9so662965pwj.13 for ; Tue, 27 Jul 2010 10:17:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:reply-to :in-reply-to:references:date:message-id:subject:from:to:cc :content-type; bh=m7CoIsuMyLOGGjrWguBYVDJhpHS83sS73rWb1ddug7s=; b=uBqLvIs9m4IAldxXS69QRiaPxKZBoDe9s8cZwr3w84kWlqTGR/rrwJf40O9YP8hDHw 6uU/PIQY4xkbbIWZmnDFtLPjD3N51ZUyjvxEDWJA0o+d5zXMzBVXMhuaoHrkIYiJt0/x JLmDMAF9PUdC8/H7VTDvxHk1rK7vdqrer5RNc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; b=u4m9oDheAKwSwbolgN7j8/BkPGuymmHM4m3KcCBqrbs3fcH0Wo4gkt77jQmqH/hBvU 9Qfpw9t5mAlImVtz6Evu3IsoGv+H0zY9CLYcaBArXdkyAJFHVyIcu86eQwl1x6l8EV34 UbF1xNJAixWVbwZmzLCHVq5LrMY5GnHKz5qaw= MIME-Version: 1.0 Received: by 10.114.59.10 with SMTP id h10mr13520595waa.194.1280251066633; Tue, 27 Jul 2010 10:17:46 -0700 (PDT) Received: by 10.114.173.9 with HTTP; Tue, 27 Jul 2010 10:17:46 -0700 (PDT) In-Reply-To: <4C4D9779.8080505@FreeBSD.org> References: <4C4AF046.40507@FreeBSD.org> <4C4B720A.6020802@FreeBSD.org> <4C4D9779.8080505@FreeBSD.org> Date: Tue, 27 Jul 2010 12:17:46 -0500 Message-ID: From: Alan Cox To: Alexander Motin X-Mailman-Approved-At: Tue, 27 Jul 2010 17:25:39 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org, Robert Watson , Rui Paulo Subject: Re: Intel TurboBoost in practice X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: alc@freebsd.org List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2010 17:17:47 -0000 On Mon, Jul 26, 2010 at 9:11 AM, Alexander Motin wrote: > Robert Watson wrote: > > On Sun, 25 Jul 2010, Alexander Motin wrote: > >>> The numbers that you are showing doesn't show much difference. Have > >>> you tried buildworld? > >> > >> If you mean relative difference -- as I have told, it's mostly because > >> of my CPU. It's maximal boost is 266MHz (8.3%), but 133MHz of them is > >> enabled most of time if CPU is not overheated. It probably doesn't, as > >> it works on clear table under air conditioner. So maximal effect I can > >> expect on is 4.2%. In such situation 2.8% probably not so bad to > >> illustrate that feature works and there is space for further > >> improvements. If I had Core i5-750S I would expect 33% boost. > > > > Can I recommend the use of ministat(1) and sample sizes of at least 8 > > runs per configuration? > > Thanks for pushing me to do it right. :) Here is 3*15 runs with fresh > kernel with disabled debug. Results are quite close to original: -2.73% > and -2.19% of time. > x C1 > + C2 > * C3 > +-----------------------------------------------------------------+ > |+ * x | > |+ * x | > |+ * x | > |+ * x | > |+ * x | > |+ * x | > |+ * x | > |+ ** x | > |+ + ** xx | > |+ + ** ** xx x| > | |__M_A____| | > |A| | > | |A| | > +-----------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 15 12.68 12.84 12.69 12.698667 0.039254966 > + 15 12.35 12.36 12.35 12.351333 0.0035186578 > Difference at 95.0% confidence > -0.347333 +/- 0.0208409 > -2.7352% +/- 0.164119% > (Student's t, pooled s = 0.0278687) > * 15 12.41 12.44 12.42 12.42 0.0075592895 > Difference at 95.0% confidence > -0.278667 +/- 0.0211391 > -2.19446% +/- 0.166467% > (Student's t, pooled s = 0.0282674) > > I also checked one more aspect -- TurboBoost works only when CPU runs at > highest EIST frequency (P0 state). I've reduced dev.cpu.0.freq from 3201 > to 3067 and repeated the test: > x C1 > + C2 > * C3 > +-----------------------------------------------------------------+ > | x + * | > | x + * | > | x + * | > | x + * *| > | x x + * *| > | x x + + * *| > | x x + + * *| > | x x + + * *| > | x x + + + + * *| > ||MA| | > | |_MA_| | > | M_A_|| > +-----------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 15 13.72 13.73 13.72 13.723333 0.0048795004 > + 15 13.79 13.82 13.8 13.803333 0.0072374686 > Difference at 95.0% confidence > 0.08 +/- 0.00461567 > 0.582949% +/- 0.0336337% > (Student's t, pooled s = 0.00617213) > * 15 13.89 13.9 13.89 13.894 0.0050709255 > Difference at 95.0% confidence > 0.170667 +/- 0.00372127 > 1.24362% +/- 0.0271164% > (Student's t, pooled s = 0.00497613) > > In that case using C2 or C3 predictably caused small performance reduce, > as after falling to sleep, CPU needs time to wakeup. Even if tested CPU0 > won't ever sleep during test, it's TLB shutdown IPIs to other cores > still probably could suffer from waiting other cores' wakeup. > > In the deeper sleep states, are the TLB contents actually maintained while the processor sleeps? (I notice that in some configurations, we actually flush dirty data from the cache before sleeping.) Alan From owner-freebsd-performance@FreeBSD.ORG Tue Jul 27 18:44:55 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A1B311065670; Tue, 27 Jul 2010 18:44:55 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id CD8508FC14; Tue, 27 Jul 2010 18:44:54 +0000 (UTC) Received: by fxm13 with SMTP id 13so861548fxm.13 for ; Tue, 27 Jul 2010 11:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=vwM3oOzazoVICoeBt30W6FdEX3ICVIxGQZyUCLrVI0s=; b=sJjGMhgZ2zl2L2B9nJke024C3vz8RdA/UNuw/0cSYa6DriYn8dta2NbRAX7HbVqBSh vH3B+ssWSVnJqS2879EkFJAQRcl6k4WAUWDZllHAuED4K+QItYATR/2jliLHnQpIEAZB FzG3/qUglHSZDf0Cvo8S8Ctd2WB43gAD+HcAQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=nvvJGTqd0X9Ud9x+5Dk85V33PEoy6uH+ZKrflOrBCZIVXXyUHv4kxktIIsxpKdnMD/ 71Ef19verD6Kt3PCYIoQyoOLZ8ELA9PzkmN1BAQ1A/ZeBQ5XTqfGEkrHFrC/Gyjt5hld u+m7kxyZqrmimtW6nc2dcVUevpDYJiF2EA+Oc= Received: by 10.223.119.131 with SMTP id z3mr8524223faq.61.1280256293689; Tue, 27 Jul 2010 11:44:53 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (pc.mavhome.dp.ua [212.86.226.226]) by mx.google.com with ESMTPS id e22sm1418129faa.0.2010.07.27.11.44.52 (version=SSLv3 cipher=RC4-MD5); Tue, 27 Jul 2010 11:44:52 -0700 (PDT) Sender: Alexander Motin Message-ID: <4C4F2921.5030604@FreeBSD.org> Date: Tue, 27 Jul 2010 21:44:49 +0300 From: Alexander Motin User-Agent: Thunderbird 2.0.0.24 (X11/20100402) MIME-Version: 1.0 To: alc@freebsd.org References: <4C4AF046.40507@FreeBSD.org> <4C4B720A.6020802@FreeBSD.org> <4C4D9779.8080505@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 27 Jul 2010 18:54:12 +0000 Cc: freebsd-hackers@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Intel TurboBoost in practice X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2010 18:44:55 -0000 Alan Cox wrote: > On Mon, Jul 26, 2010 at 9:11 AM, Alexander Motin > wrote: > > In that case using C2 or C3 predictably caused small performance reduce, > as after falling to sleep, CPU needs time to wakeup. Even if tested CPU0 > won't ever sleep during test, it's TLB shutdown IPIs to other cores > still probably could suffer from waiting other cores' wakeup. > > In the deeper sleep states, are the TLB contents actually maintained > while the processor sleeps? (I notice that in some configurations, we > actually flush dirty data from the cache before sleeping.) As I understand, we flush caches only as last resort, if platform does not supports special techniques, such as disabling arbitration or making CPU to wake up on bus mastering. But same ACPI C-states could map into different CPU C-states. Some of these CPU states (like C6) could imply caches invalidation, though I am not sure it can be seen outside. ACPI 3.0 specification tells nothing about TLBs, so I am not sure we can count on their invalidation, except we do it ourselves, like it is done for caches when CPU can't keep their coherency while sleeping. -- Alexander Motin From owner-freebsd-performance@FreeBSD.ORG Wed Jul 28 13:48:29 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5DA5106566B for ; Wed, 28 Jul 2010 13:48:29 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by mx1.freebsd.org (Postfix) with ESMTP id 7A6AD8FC12 for ; Wed, 28 Jul 2010 13:48:29 +0000 (UTC) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.14.4/8.14.3) with ESMTP id o6SDRoZh022359 for ; Wed, 28 Jul 2010 09:27:50 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <201007281327.o6SDRoZh022359@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 28 Jul 2010 09:27:48 -0400 To: freebsd-performance@freebsd.org From: Mike Tancsa Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: tuning zfs for large file reads X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2010 13:48:30 -0000 Every once in a while I need to go through some rather large netflow/argus logs (each file about 1G, total of about 90 files) stored on a zfs array and I am finding reading from the file is at best about 50MB/s and often much worse. The individual underlying disks should be able to read much faster than that I would think. Is there anything I can do to tune large file read performance a bit more ? I do have compression on, so not sure how much that is impacting things. I do plan to add more storage soon, so perhaps another 4 spindles might help to spread the load ? e.g. # iostat -c 100 ada0 ada1 ada2 ada3 tty ada0 ada1 ada2 ada3 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 24 34.51 70 2.37 38.64 56 2.10 38.60 65 2.45 38.10 56 2.09 5 0 4 0 91 0 279 28.11 397 10.89 29.65 251 7.26 36.14 285 10.06 30.03 250 7.33 17 0 10 1 72 0 93 27.66 361 9.74 30.23 201 5.93 35.60 275 9.56 30.31 223 6.61 12 0 8 0 79 0 94 23.27 345 7.83 32.51 187 5.94 40.10 250 9.79 29.80 173 5.04 14 0 13 0 73 0 94 28.59 326 9.10 29.10 234 6.64 37.00 235 8.48 28.53 198 5.52 12 0 25 0 63 0 94 21.23 420 8.72 23.88 237 5.52 33.71 285 9.39 26.91 239 6.28 38 0 18 1 43 0 94 27.24 352 9.37 24.60 238 5.71 33.94 275 9.12 26.99 235 6.19 23 0 19 0 59 0 93 30.95 223 6.75 35.46 160 5.55 37.01 172 6.22 31.46 147 4.50 7 0 7 0 85 0 93 23.55 349 8.02 30.52 194 5.77 33.86 276 9.13 28.82 216 6.09 16 0 9 0 75 0 94 33.06 350 11.29 44.79 167 7.32 45.95 230 10.30 39.10 137 5.23 14 0 12 0 75 0 93 25.32 254 6.28 29.80 147 4.27 31.92 225 7.02 27.21 175 4.65 8 0 8 0 84 0 93 26.92 380 9.99 28.80 247 6.95 34.63 299 10.11 27.18 273 7.24 15 0 11 0 74 0 93 25.77 287 7.23 22.90 179 4.01 31.63 241 7.45 22.71 202 4.48 13 0 14 0 73 gstat shows the disks 100% busy while doing the read # zfs get all NAME PROPERTY VALUE SOURCE zbackup1 type filesystem - zbackup1 creation Tue Apr 29 22:48 2008 - zbackup1 used 1.91T - zbackup1 available 781G - zbackup1 referenced 1.91T - zbackup1 compressratio 1.43x - zbackup1 mounted yes - zbackup1 quota none default zbackup1 reservation none default zbackup1 recordsize 128K default zbackup1 mountpoint /zbackup1 default zbackup1 sharenfs off default zbackup1 checksum on default zbackup1 compression on local zbackup1 atime off local zbackup1 devices on default zbackup1 exec on default zbackup1 setuid on default zbackup1 readonly off default zbackup1 jailed off default zbackup1 snapdir hidden default zbackup1 aclmode groupmask default zbackup1 aclinherit restricted default zbackup1 canmount on default zbackup1 shareiscsi off default zbackup1 xattr off temporary zbackup1 copies 1 default zbackup1 version 3 - zbackup1 utf8only off - zbackup1 normalization none - zbackup1 casesensitivity sensitive - zbackup1 vscan off default zbackup1 nbmand off default zbackup1 sharesmb off default zbackup1 refquota none default zbackup1 refreservation none default zbackup1 primarycache all default zbackup1 secondarycache all default # zpool status pool: zbackup1 state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM zbackup1 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada0 ONLINE 0 0 0 errors: No known data errors ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich3 bus 0 scbus7 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada2 at ahcich4 bus 0 scbus8 target 0 lun 0 ada2: ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada3 at ahcich5 bus 0 scbus9 target 0 lun 0 ada3: ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) Its an intel ICH10 based chipset. Its RELENG_8 from July 13th, AMD64 8gig of RAM vfs.zfs.l2c_only_size: 0 vfs.zfs.mfu_ghost_data_lsize: 8519680 vfs.zfs.mfu_ghost_metadata_lsize: 95119872 vfs.zfs.mfu_ghost_size: 103639552 vfs.zfs.mfu_data_lsize: 28704768 vfs.zfs.mfu_metadata_lsize: 33280 vfs.zfs.mfu_size: 60260864 vfs.zfs.mru_ghost_data_lsize: 5898240 vfs.zfs.mru_ghost_metadata_lsize: 106185728 vfs.zfs.mru_ghost_size: 112083968 vfs.zfs.mru_data_lsize: 33816576 vfs.zfs.mru_metadata_lsize: 32736768 vfs.zfs.mru_size: 77434368 vfs.zfs.anon_data_lsize: 0 vfs.zfs.anon_metadata_lsize: 0 vfs.zfs.anon_size: 2245120 vfs.zfs.l2arc_norw: 1 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_noprefetch: 0 vfs.zfs.l2arc_feed_min_ms: 200 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_headroom: 2 vfs.zfs.l2arc_write_boost: 8388608 vfs.zfs.l2arc_write_max: 8388608 vfs.zfs.arc_meta_limit: 432819840 vfs.zfs.arc_meta_used: 151634344 vfs.zfs.mdcomp_disable: 0 vfs.zfs.arc_min: 216409920 vfs.zfs.arc_max: 1731279360 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.block_cap: 256 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 0 vfs.zfs.check_hostid: 1 vfs.zfs.recover: 0 vfs.zfs.txg.write_limit_override: 0 vfs.zfs.txg.synctime: 5 vfs.zfs.txg.timeout: 30 vfs.zfs.scrub_limit: 10 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.ramp_rate: 2 vfs.zfs.vdev.time_shift: 6 vfs.zfs.vdev.min_pending: 4 vfs.zfs.vdev.max_pending: 35 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_disable: 0 vfs.zfs.zio.use_uma: 0 vfs.zfs.version.zpl: 3 vfs.zfs.version.vdev_boot: 1 vfs.zfs.version.spa: 14 vfs.zfs.version.dmu_backup_stream: 1 vfs.zfs.version.dmu_backup_header: 2 vfs.zfs.version.acl: 1 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 kstat.zfs.misc.zfetchstats.hits: 3276257635 kstat.zfs.misc.zfetchstats.misses: 358569869 kstat.zfs.misc.zfetchstats.colinear_hits: 101066 kstat.zfs.misc.zfetchstats.colinear_misses: 358468803 kstat.zfs.misc.zfetchstats.stride_hits: 3232647654 kstat.zfs.misc.zfetchstats.stride_misses: 32579337 kstat.zfs.misc.zfetchstats.reclaim_successes: 3639515 kstat.zfs.misc.zfetchstats.reclaim_failures: 354829288 kstat.zfs.misc.zfetchstats.streams_resets: 547669 kstat.zfs.misc.zfetchstats.streams_noresets: 43427240 kstat.zfs.misc.zfetchstats.bogus_streams: 0 kstat.zfs.misc.arcstats.hits: 1628471537 kstat.zfs.misc.arcstats.misses: 157059604 kstat.zfs.misc.arcstats.demand_data_hits: 56872510 kstat.zfs.misc.arcstats.demand_data_misses: 7357670 kstat.zfs.misc.arcstats.demand_metadata_hits: 811953750 kstat.zfs.misc.arcstats.demand_metadata_misses: 92175332 kstat.zfs.misc.arcstats.prefetch_data_hits: 21867244 kstat.zfs.misc.arcstats.prefetch_data_misses: 35428145 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 737778033 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 22098457 kstat.zfs.misc.arcstats.mru_hits: 255443336 kstat.zfs.misc.arcstats.mru_ghost_hits: 37264902 kstat.zfs.misc.arcstats.mfu_hits: 614687158 kstat.zfs.misc.arcstats.mfu_ghost_hits: 28488139 kstat.zfs.misc.arcstats.allocated: 178693054 kstat.zfs.misc.arcstats.deleted: 100291619 kstat.zfs.misc.arcstats.stolen: 32463801 kstat.zfs.misc.arcstats.recycle_miss: 112803117 kstat.zfs.misc.arcstats.mutex_miss: 595987 kstat.zfs.misc.arcstats.evict_skip: 43698367555 kstat.zfs.misc.arcstats.evict_l2_cached: 0 kstat.zfs.misc.arcstats.evict_l2_eligible: 4878982700544 kstat.zfs.misc.arcstats.evict_l2_ineligible: 1884938647552 kstat.zfs.misc.arcstats.hash_elements: 47690 kstat.zfs.misc.arcstats.hash_elements_max: 211948 kstat.zfs.misc.arcstats.hash_collisions: 23698456 kstat.zfs.misc.arcstats.hash_chains: 8682 kstat.zfs.misc.arcstats.hash_chain_max: 30 kstat.zfs.misc.arcstats.p: 202873036 kstat.zfs.misc.arcstats.c: 216409920 kstat.zfs.misc.arcstats.c_min: 216409920 kstat.zfs.misc.arcstats.c_max: 1731279360 kstat.zfs.misc.arcstats.size: 216359384 kstat.zfs.misc.arcstats.hdr_size: 10358144 kstat.zfs.misc.arcstats.data_size: 139796480 kstat.zfs.misc.arcstats.other_size: 66204760 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_read_bytes: 0 kstat.zfs.misc.arcstats.l2_write_bytes: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 21505931 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 kstat.zfs.misc.arcstats.l2_write_in_l2: 0 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 26447966 kstat.zfs.misc.arcstats.l2_write_full: 0 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0 kstat.zfs.misc.arcstats.l2_write_pios: 0 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0 kstat.zfs.misc.vdev_cache_stats.delegations: 14308501 kstat.zfs.misc.vdev_cache_stats.hits: 145613677 kstat.zfs.misc.vdev_cache_stats.misses: 22833085 some stats during a read. # sysctl -a kstat.zfs.misc.arcstats;sleep 5;sysctl kstat.zfs.misc.arcstats kstat.zfs.misc.arcstats.hits: 1628766598 kstat.zfs.misc.arcstats.misses: 157095912 kstat.zfs.misc.arcstats.demand_data_hits: 57123742 kstat.zfs.misc.arcstats.demand_data_misses: 7361404 kstat.zfs.misc.arcstats.demand_metadata_hits: 811993421 kstat.zfs.misc.arcstats.demand_metadata_misses: 92177322 kstat.zfs.misc.arcstats.prefetch_data_hits: 21870856 kstat.zfs.misc.arcstats.prefetch_data_misses: 35458138 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 737778579 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 22099048 kstat.zfs.misc.arcstats.mru_hits: 255507575 kstat.zfs.misc.arcstats.mru_ghost_hits: 37268564 kstat.zfs.misc.arcstats.mfu_hits: 614913822 kstat.zfs.misc.arcstats.mfu_ghost_hits: 28489531 kstat.zfs.misc.arcstats.allocated: 178729987 kstat.zfs.misc.arcstats.deleted: 100325316 kstat.zfs.misc.arcstats.stolen: 32480168 kstat.zfs.misc.arcstats.recycle_miss: 112813638 kstat.zfs.misc.arcstats.mutex_miss: 596031 kstat.zfs.misc.arcstats.evict_skip: 43705615686 kstat.zfs.misc.arcstats.evict_l2_cached: 0 kstat.zfs.misc.arcstats.evict_l2_eligible: 4882493084160 kstat.zfs.misc.arcstats.evict_l2_ineligible: 1885466736640 kstat.zfs.misc.arcstats.hash_elements: 45389 kstat.zfs.misc.arcstats.hash_elements_max: 211948 kstat.zfs.misc.arcstats.hash_collisions: 23703581 kstat.zfs.misc.arcstats.hash_chains: 7986 kstat.zfs.misc.arcstats.hash_chain_max: 30 kstat.zfs.misc.arcstats.p: 202884300 kstat.zfs.misc.arcstats.c: 216409920 kstat.zfs.misc.arcstats.c_min: 216409920 kstat.zfs.misc.arcstats.c_max: 1731279360 kstat.zfs.misc.arcstats.size: 217163440 kstat.zfs.misc.arcstats.hdr_size: 9940256 kstat.zfs.misc.arcstats.data_size: 135882240 kstat.zfs.misc.arcstats.other_size: 71340944 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_read_bytes: 0 kstat.zfs.misc.arcstats.l2_write_bytes: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 21731398 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 kstat.zfs.misc.arcstats.l2_write_in_l2: 0 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 26451995 kstat.zfs.misc.arcstats.l2_write_full: 0 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0 kstat.zfs.misc.arcstats.l2_write_pios: 0 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0 kstat.zfs.misc.arcstats.hits: 1628769047 kstat.zfs.misc.arcstats.misses: 157097445 kstat.zfs.misc.arcstats.demand_data_hits: 57124342 kstat.zfs.misc.arcstats.demand_data_misses: 7361439 kstat.zfs.misc.arcstats.demand_metadata_hits: 811995060 kstat.zfs.misc.arcstats.demand_metadata_misses: 92177359 kstat.zfs.misc.arcstats.prefetch_data_hits: 21871065 kstat.zfs.misc.arcstats.prefetch_data_misses: 35459599 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 737778580 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 22099048 kstat.zfs.misc.arcstats.mru_hits: 255508558 kstat.zfs.misc.arcstats.mru_ghost_hits: 37269451 kstat.zfs.misc.arcstats.mfu_hits: 614915078 kstat.zfs.misc.arcstats.mfu_ghost_hits: 28489553 kstat.zfs.misc.arcstats.allocated: 178731542 kstat.zfs.misc.arcstats.deleted: 100325916 kstat.zfs.misc.arcstats.stolen: 32481046 kstat.zfs.misc.arcstats.recycle_miss: 112813939 kstat.zfs.misc.arcstats.mutex_miss: 596033 kstat.zfs.misc.arcstats.evict_skip: 43705791814 kstat.zfs.misc.arcstats.evict_l2_cached: 0 kstat.zfs.misc.arcstats.evict_l2_eligible: 4882575294464 kstat.zfs.misc.arcstats.evict_l2_ineligible: 1885578672128 kstat.zfs.misc.arcstats.hash_elements: 45419 kstat.zfs.misc.arcstats.hash_elements_max: 211948 kstat.zfs.misc.arcstats.hash_collisions: 23703681 kstat.zfs.misc.arcstats.hash_chains: 7991 kstat.zfs.misc.arcstats.hash_chain_max: 30 kstat.zfs.misc.arcstats.p: 202884300 kstat.zfs.misc.arcstats.c: 216409920 kstat.zfs.misc.arcstats.c_min: 216409920 kstat.zfs.misc.arcstats.c_max: 1731279360 kstat.zfs.misc.arcstats.size: 216364904 kstat.zfs.misc.arcstats.hdr_size: 9946424 kstat.zfs.misc.arcstats.data_size: 135031808 kstat.zfs.misc.arcstats.other_size: 71386672 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_read_bytes: 0 kstat.zfs.misc.arcstats.l2_write_bytes: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 21731407 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 kstat.zfs.misc.arcstats.l2_write_in_l2: 0 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 26452849 kstat.zfs.misc.arcstats.l2_write_full: 0 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0 kstat.zfs.misc.arcstats.l2_write_pios: 0 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0 -------------------------------------------------------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet since 1994 www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike From owner-freebsd-performance@FreeBSD.ORG Wed Jul 28 14:14:22 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EF091065752 for ; Wed, 28 Jul 2010 14:14:22 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 7BA8B8FC15 for ; Wed, 28 Jul 2010 14:14:21 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA05012; Wed, 28 Jul 2010 17:01:05 +0300 (EEST) (envelope-from avg@icyb.net.ua) Message-ID: <4C503820.5080106@icyb.net.ua> Date: Wed, 28 Jul 2010 17:01:04 +0300 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.24 (X11/20100517) MIME-Version: 1.0 To: Mike Tancsa References: <201007281327.o6SDRoZh022359@lava.sentex.ca> In-Reply-To: <201007281327.o6SDRoZh022359@lava.sentex.ca> X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 28 Jul 2010 15:38:36 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: tuning zfs for large file reads X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2010 14:14:22 -0000 Can't really help you but here is couple of notes: 1. iostat -x output seems to be even more informative for disk I/O 2. Despite large amount of RAM (8GB) your ZFS ARC size stays at its minimum value near just 200MB. Not sure if this plays significant role in the performance issue, but perhaps something to look at. -- Andriy Gapon