From owner-freebsd-arch@FreeBSD.ORG Sat Mar 20 21:50:12 2010 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E8F81065675 for ; Sat, 20 Mar 2010 21:50:12 +0000 (UTC) (envelope-from freebsd-arch@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id EC1AD8FC14 for ; Sat, 20 Mar 2010 21:50:11 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Nt6Yc-0002C6-AD for freebsd-arch@freebsd.org; Sat, 20 Mar 2010 22:50:10 +0100 Received: from 93-138-28-199.adsl.net.t-com.hr ([93.138.28.199]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 20 Mar 2010 22:50:10 +0100 Received: from ivoras by 93-138-28-199.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 20 Mar 2010 22:50:10 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-arch@freebsd.org From: Ivan Voras Date: Sat, 20 Mar 2010 22:49:53 +0100 Lines: 44 Message-ID: References: <4BA4E7A9.3070502@FreeBSD.org> <201003201753.o2KHrH5x003946@apollo.backplane.com> <891E2580-8DE3-4B82-81C4-F2C07735A854@samsco.org> <4BA52179.9030903@FreeBSD.org> <4BA532FF.6040407@elischer.org> <4BA53E77.8010208@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 93-138-28-199.adsl.net.t-com.hr User-Agent: Thunderbird 2.0.0.21 (X11/20090612) In-Reply-To: <4BA53E77.8010208@elischer.org> Cc: freebsd-current@freebsd.org Subject: Re: Increasing MAXPHYS X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Mar 2010 21:50:12 -0000 Julian Elischer wrote: > Ivan Voras wrote: >> Julian Elischer wrote: >>> Alexander Motin wrote: >>>> Scott Long wrote: >>>>> On Mar 20, 2010, at 11:53 AM, Matthew Dillon wrote: >>>>>> Diminishing returns get hit pretty quickly with larger MAXPHYS >>>>>> values. >>>>>> As long as the I/O can be pipelined the reduced transaction rate >>>>>> becomes less interesting when the transaction rate is less than a >>>>>> certain level. Off the cuff I'd say 2000 tps is a good basis for >>>>>> considering whether it is an issue or not. 256K is actually quite >>>>>> a reasonable value. Even 128K is reasonable. >>>>> I agree completely. I did quite a bit of testing on this in 2008 >>>>> and 2009. >>>>> I even added some hooks into CAM to support this, and I thought >>>>> that I had >>>>> discussed this extensively with Alexander at the time. Guess it >>>>> was yet another >>>>> wasted conversation with him =-( I'll repeat it here for the record. >>> >>> In the Fusion-io driver we find that the limiting factor is not the >>> size of MAXPHYS, but the fact that we can not push more than >>> 170k tps through geom. (in my test machine. I've seen more on some >>> beefier machines), but that is only a limit on small transacrtions, >> >> Do the GEOM threads (g_up, g_down) go into saturation? Effectively all >> IO is serialized through them. > > basically.. > > You can get better throughput by using TSC for timing because the geom > and devstat code does a bit of timing.. Geom can be told to turn off > it's timing but devstat can't. The 170 ktps is with TSC as timer, > and geom timing turned off. I see. I just ran randomio on a gzero device and with 10 userland threads (this is a slow 2xquad machine) I get g_up and g_down saturated fast with ~~ 120 ktps. Randomio uses gettimeofday() for measurements. Hmm, it looks like it could be easy to spawn more g_* threads (and, barring specific class behaviour, it has a fair chance of working out of the box) but the incoming queue will need to also be broken up for greater effect.