From owner-freebsd-stable@freebsd.org Sun Mar 6 21:16:27 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4B13A95432 for ; Sun, 6 Mar 2016 21:16:27 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from mail.samsco.org (suzy.samsco.org [168.103.85.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F607882 for ; Sun, 6 Mar 2016 21:16:25 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from localhost (localhost [192.168.254.3]) by mail.samsco.org (Postfix) with ESMTP id ED38415C1867E; Sun, 6 Mar 2016 21:12:42 +0000 (UTC) Received: from mail.samsco.org ([192.168.254.3]) by localhost (mail.samsco.org [192.168.254.3]) (maiad, port 10024) with ESMTP id 70166-03; Sun, 6 Mar 2016 21:12:42 +0000 (UTC) Received: from [10.12.11.125] (ip-64-134-226-15.public.wayport.net [64.134.226.15]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: scottl@samsco.org) by mail.samsco.org (Postfix) with ESMTPSA id 3968B15C1867C; Sun, 6 Mar 2016 21:12:42 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains. From: Scott Long X-Mailer: iPhone Mail (13D15) In-Reply-To: <20160306194555.GC94639@zxy.spb.ru> Date: Sun, 6 Mar 2016 13:10:42 -0800 Cc: freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0F0C78F4-6FE2-43BA-B503-AA04A79F2E70@samsco.org> References: <20160306194555.GC94639@zxy.spb.ru> To: Slawa Olhovchenkov X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2016 21:16:27 -0000 Hi, The message is harmless, it's a reminder that you should tune the kernel for= your workload. When the message is triggered, it means that a potential co= mmand was deferred, likely for only a few microseconds, and then everything m= oved on as normal. =20 A command uses anywhere from 0 to a few dozen chain frames per I/o, dependin= g on the size of the io. The chain frame memory is allocated at boot so tha= t it's always available, not allocated on the fly. When I wrote this driver= , I felt that it would be wasteful to reserve memory for a worst case scenar= io of all large io's by default, so I put in this deferral system with a con= sole reminder to for tuning. =20 Yes, you actually do have 900 io's outstanding. The controller buffers the i= o requests and allows the system to queue up much more than what sata disks m= ight allow on their own. It's debatable if this is good or bad, but it's tu= nable as well. Anyways, the messages should not cause alarm. Either tune up the chain fram= e count, or tune down the max io count. Scott Sent from my iPhone > On Mar 6, 2016, at 11:45 AM, Slawa Olhovchenkov wrote: >=20 > I am use 10-STABLE r295539 and LSI SAS2008. >=20 > mps0: port 0x8000-0x80ff mem 0xdfc00000= -0xdfc03fff,0xdfb80000-0xdfbbffff irq 32 at device 0.0 on pci2 > mps0: Firmware: 15.00.00.00, Driver: 20.00.00.00-fbsd > mps0: IOCCapabilities: 185c > mps1: port 0x7000-0x70ff mem 0xdf600000= -0xdf603fff,0xdf580000-0xdf5bffff irq 34 at device 0.0 on pci3 > mps1: Firmware: 17.00.01.00, Driver: 20.00.00.00-fbsd > mps1: IOCCapabilities: 185c > mps2: port 0xf000-0xf0ff mem 0xfba00000= -0xfba03fff,0xfb980000-0xfb9bffff irq 50 at device 0.0 on pci129 > mps2: Firmware: 15.00.00.00, Driver: 20.00.00.00-fbsd > mps2: IOCCapabilities: 185c > mps3: port 0xe000-0xe0ff mem 0xfb400000= -0xfb403fff,0xfb380000-0xfb3bffff irq 56 at device 0.0 on pci130 > mps3: Firmware: 15.00.00.00, Driver: 20.00.00.00-fbsd > mps3: IOCCapabilities: 185c >=20 > Some time ago I am see in log messages like this: >=20 > Mar 6 22:28:27 edge02 kernel: mps3: Out of chain frames, consider increas= ing hw.mps.max_chains. > Mar 6 22:28:28 edge02 kernel: mps1: Out of chain frames, consider increas= ing hw.mps.max_chains. > Mar 6 22:28:28 edge02 kernel: mps0: Out of chain frames, consider increas= ing hw.mps.max_chains. > Mar 6 22:29:39 edge02 kernel: mps0: Out of chain frames, consider increas= ing hw.mps.max_chains. > Mar 6 22:30:07 edge02 kernel: mps3: Out of chain frames, consider increas= ing hw.mps.max_chains. > Mar 6 22:30:09 edge02 kernel: mps1: Out of chain frames, consider increas= ing hw.mps.max_chains. >=20 > This is peak hour. I am try to monitoring: >=20 > root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active > dev.mps.3.chain_free: 70 > dev.mps.3.io_cmds_active: 901 > dev.mps.2.chain_free: 504 > dev.mps.2.io_cmds_active: 725 > dev.mps.1.chain_free: 416 > dev.mps.1.io_cmds_active: 896 > dev.mps.0.chain_free: 39 > dev.mps.0.io_cmds_active: 12 > root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active > dev.mps.3.chain_free: 412 > dev.mps.3.io_cmds_active: 572 > dev.mps.2.chain_free: 718 > dev.mps.2.io_cmds_active: 687 > dev.mps.1.chain_free: 211 > dev.mps.1.io_cmds_active: 906 > dev.mps.0.chain_free: 65 > dev.mps.0.io_cmds_active: 144 > root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active > dev.mps.3.chain_free: 500 > dev.mps.3.io_cmds_active: 629 > dev.mps.2.chain_free: 623 > dev.mps.2.io_cmds_active: 676 > dev.mps.1.chain_free: 251 > dev.mps.1.io_cmds_active: 907 > dev.mps.0.chain_free: 139 > dev.mps.0.io_cmds_active: 144 >=20 > [...] >=20 > root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active > dev.mps.3.chain_free: 1874 > dev.mps.3.io_cmds_active: 78 > dev.mps.2.chain_free: 1888 > dev.mps.2.io_cmds_active: 64 > dev.mps.1.chain_free: 1922 > dev.mps.1.io_cmds_active: 42 > dev.mps.0.chain_free: 1936 > dev.mps.0.io_cmds_active: 48 > root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active > dev.mps.3.chain_free: 1890 > dev.mps.3.io_cmds_active: 78 > dev.mps.2.chain_free: 1890 > dev.mps.2.io_cmds_active: 82 > dev.mps.1.chain_free: 1729 > dev.mps.1.io_cmds_active: 150 > dev.mps.0.chain_free: 1893 > dev.mps.0.io_cmds_active: 57 >=20 > What this mean? Why with allocated 2048 chains per controller I see 65 fre= e and 144 allocated? > How I got 976 active commands? I am use SATA HDD suported only 32 tags on N= CQ, with 8 ports > this maximum 256 outstanding commands per controller. >=20 > How I can resolve this issue? > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"