From owner-freebsd-stable@freebsd.org Mon Mar 7 07:40:59 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CFCB9AC1CCA for ; Mon, 7 Mar 2016 07:40:59 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from mail.samsco.org (suzy.samsco.org [168.103.85.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8EB29A69 for ; Mon, 7 Mar 2016 07:40:58 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from localhost (localhost [192.168.254.3]) by mail.samsco.org (Postfix) with ESMTP id CD5D915C1867E; Mon, 7 Mar 2016 07:42:49 +0000 (UTC) Received: from mail.samsco.org ([192.168.254.3]) by localhost (mail.samsco.org [192.168.254.3]) (maiad, port 10024) with ESMTP id 80111-05; Mon, 7 Mar 2016 07:42:49 +0000 (UTC) Received: from [10.81.36.75] (75-144-23-245-SFBACA.hfc.comcastbusiness.net [75.144.23.245]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: scottl@samsco.org) by mail.samsco.org (Postfix) with ESMTPSA id 50DE815C1867C; Mon, 7 Mar 2016 07:42:49 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains. From: Scott Long In-Reply-To: <20160307060407.GK11654@zxy.spb.ru> Date: Sun, 6 Mar 2016 23:40:55 -0800 Cc: freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <5B8DD95A-9FA0-4E16-85A1-87B54035B3F7@samsco.org> References: <20160306194555.GC94639@zxy.spb.ru> <0F0C78F4-6FE2-43BA-B503-AA04A79F2E70@samsco.org> <20160306212733.GJ11654@zxy.spb.ru> <20160307060407.GK11654@zxy.spb.ru> To: Slawa Olhovchenkov X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2016 07:41:00 -0000 > On Mar 6, 2016, at 10:04 PM, Slawa Olhovchenkov = wrote: >=20 > On Sun, Mar 06, 2016 at 06:20:06PM -0800, Scott Long wrote: >=20 >>=20 >>> On Mar 6, 2016, at 1:27 PM, Slawa Olhovchenkov = wrote: >>>=20 >>> On Sun, Mar 06, 2016 at 01:10:42PM -0800, Scott Long wrote: >>>=20 >>>> Hi, >>>>=20 >>>> The message is harmless, it's a reminder that you should tune the = kernel for your workload. When the message is triggered, it means that = a potential command was deferred, likely for only a few microseconds, = and then everything moved on as normal. =20 >>>>=20 >>>> A command uses anywhere from 0 to a few dozen chain frames per I/o, = depending on the size of the io. The chain frame memory is allocated at = boot so that it's always available, not allocated on the fly. When I = wrote this driver, I felt that it would be wasteful to reserve memory = for a worst case scenario of all large io's by default, so I put in this = deferral system with a console reminder to for tuning. =20 >>>>=20 >>>> Yes, you actually do have 900 io's outstanding. The controller = buffers the io requests and allows the system to queue up much more than = what sata disks might allow on their own. It's debatable if this is = good or bad, but it's tunable as well. >>>>=20 >>>> Anyways, the messages should not cause alarm. Either tune up the = chain frame count, or tune down the max io count. >>>=20 >>> I am don't know depends or not, but I see dramaticaly performance = drop >>> at time of this messages. >>>=20 >>=20 >> Good to know. Part of the performance drop might be because of the = slowness of printing to the console. >=20 > no, on console print may be one per minute >=20 The one-per-minute prints are by design. I should probably make it = print once and then increment a sysctl counter. >>> How I can calculate buffers numbers? >>=20 >> If your system is new enough to have mpsutil, please run it = =E2=80=98mpsutil >> show iocfacts=E2=80=99. >=20 > As I see mpsutil present only on -HEAD. > Can I compile it on 10-STABLE? >=20 Yes, I believe it should compile on 10, but I have not tried it = recently. >> If not, then boot your system with bootverbose and send me the = output. >=20 > I can do this day ago. >=20 >>> I am have very heavy I/O. >>=20 >> Out of curiosity, do you redefine MAXPHYS/DFLTPHYS in your kernel = config? >=20 > no >=20 >>> This allocated one for all controllers, or allocated for every = controller? >>=20 >> It=E2=80=99s per-controller. >>=20 >> I=E2=80=99ve thought about making the tuning be dynamic at runtime. = I >> implemented similar dynamic tuning for other drivers, but it seemed >> overly complex for low benefit. Implementing it for this driver >> would be possible but require some significant code changes. >=20 > What cause of chain_free+io_cmds_active << max_chains? > One cmd can use many chains? Yes. A request uses and active command, and depending on the size of = the I/O, it might use several chain frames. Scott