From owner-freebsd-stable@freebsd.org Tue Mar 8 19:02:08 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7F3DAC8361 for ; Tue, 8 Mar 2016 19:02:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6BBA31458 for ; Tue, 8 Mar 2016 19:02:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1adMtR-000Kyb-4n; Tue, 08 Mar 2016 22:02:05 +0300 Date: Tue, 8 Mar 2016 22:02:05 +0300 From: Slawa Olhovchenkov To: Scott Long Cc: freebsd-stable@freebsd.org Subject: Re: kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains. Message-ID: <20160308190205.GG70809@zxy.spb.ru> References: <0F0C78F4-6FE2-43BA-B503-AA04A79F2E70@samsco.org> <20160306212733.GJ11654@zxy.spb.ru> <20160307060407.GK11654@zxy.spb.ru> <5B8DD95A-9FA0-4E16-85A1-87B54035B3F7@samsco.org> <20160307111012.GL11654@zxy.spb.ru> <20160308180746.GE70809@zxy.spb.ru> <6189E959-3489-438E-8D91-9E5E46E2D482@samsco.org> <20160308184823.GF70809@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Mar 2016 19:02:08 -0000 On Tue, Mar 08, 2016 at 10:56:39AM -0800, Scott Long wrote: > > > On Mar 8, 2016, at 10:48 AM, Slawa Olhovchenkov wrote: > > > > On Tue, Mar 08, 2016 at 10:34:23AM -0800, Scott Long wrote: > > > >> > >>> On Mar 8, 2016, at 10:07 AM, Slawa Olhovchenkov wrote: > >>> > >>> On Mon, Mar 07, 2016 at 02:10:12PM +0300, Slawa Olhovchenkov wrote: > >>> > >>>>>>>> This allocated one for all controllers, or allocated for every controller? > >>>>>>> > >>>>>>> It’s per-controller. > >>>>>>> > >>>>>>> I’ve thought about making the tuning be dynamic at runtime. I > >>>>>>> implemented similar dynamic tuning for other drivers, but it seemed > >>>>>>> overly complex for low benefit. Implementing it for this driver > >>>>>>> would be possible but require some significant code changes. > >>>>>> > >>>>>> What cause of chain_free+io_cmds_active << max_chains? > >>>>>> One cmd can use many chains? > >>>>> > >>>>> Yes. A request uses and active command, and depending on the size of the I/O, > >>>>> it might use several chain frames. > >>> > >>> I am play with max_chains and like significant cost of handling > >>> max_chains: with 8192 system resonded badly vs 2048. Now try 3192, > >>> response like with 2048. > >> > >> Hi, I’m not sure I understand what you’re saying. You said that you tried 8192, but the system still complained of being out of chain frames? Now you are trying fewer, only 3192? > > > > With 8192 system not complained of being out of chain frames, but like > > need more CPU power to handle this chain list -- traffic graf (this > > host servered HTTP by nginx) have many "jerking", with 3192 traffic > > graf is more smooth. > > Hi, > > The CPU overhead of doing more chain frames is nil. They are just > objects in a list, and processing the list is O(1), not O(n). What > you are likely seeing is other problems with VM and VFS-BIO system > struggling to deal with the amount of I/O that you are doing. > Depending on what kind I/O you are doing (buffered filesystem > reads/writes, memory mapped I/O, unbuffered I/O) there are limits > and high/low water marks on how much I/O can be outstanding, and > when the limits are reached processes are put to sleep and then race > back in when they are woken up. This causes poor, oscillating > system behavior. There’s some tuning you can do to increase the > limits, but yes, it’s a problem that behaves poorly in an untuned > system. Sorry, I am don't understund you point: how to large unused chain frames can consume CPU power?