From owner-freebsd-stable@freebsd.org Sun Mar 6 19:46:05 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAC11A95C3B for ; Sun, 6 Mar 2016 19:46:05 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8F0317C2 for ; Sun, 6 Mar 2016 19:46:05 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1acecl-0008xY-DQ for freebsd-stable@freebsd.org; Sun, 06 Mar 2016 22:45:55 +0300 Date: Sun, 6 Mar 2016 22:45:55 +0300 From: Slawa Olhovchenkov To: freebsd-stable@freebsd.org Subject: kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains. Message-ID: <20160306194555.GC94639@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2016 19:46:06 -0000 I am use 10-STABLE r295539 and LSI SAS2008. mps0: port 0x8000-0x80ff mem 0xdfc00000-0xdfc03fff,0xdfb80000-0xdfbbffff irq 32 at device 0.0 on pci2 mps0: Firmware: 15.00.00.00, Driver: 20.00.00.00-fbsd mps0: IOCCapabilities: 185c mps1: port 0x7000-0x70ff mem 0xdf600000-0xdf603fff,0xdf580000-0xdf5bffff irq 34 at device 0.0 on pci3 mps1: Firmware: 17.00.01.00, Driver: 20.00.00.00-fbsd mps1: IOCCapabilities: 185c mps2: port 0xf000-0xf0ff mem 0xfba00000-0xfba03fff,0xfb980000-0xfb9bffff irq 50 at device 0.0 on pci129 mps2: Firmware: 15.00.00.00, Driver: 20.00.00.00-fbsd mps2: IOCCapabilities: 185c mps3: port 0xe000-0xe0ff mem 0xfb400000-0xfb403fff,0xfb380000-0xfb3bffff irq 56 at device 0.0 on pci130 mps3: Firmware: 15.00.00.00, Driver: 20.00.00.00-fbsd mps3: IOCCapabilities: 185c Some time ago I am see in log messages like this: Mar 6 22:28:27 edge02 kernel: mps3: Out of chain frames, consider increasing hw.mps.max_chains. Mar 6 22:28:28 edge02 kernel: mps1: Out of chain frames, consider increasing hw.mps.max_chains. Mar 6 22:28:28 edge02 kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains. Mar 6 22:29:39 edge02 kernel: mps0: Out of chain frames, consider increasing hw.mps.max_chains. Mar 6 22:30:07 edge02 kernel: mps3: Out of chain frames, consider increasing hw.mps.max_chains. Mar 6 22:30:09 edge02 kernel: mps1: Out of chain frames, consider increasing hw.mps.max_chains. This is peak hour. I am try to monitoring: root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active dev.mps.3.chain_free: 70 dev.mps.3.io_cmds_active: 901 dev.mps.2.chain_free: 504 dev.mps.2.io_cmds_active: 725 dev.mps.1.chain_free: 416 dev.mps.1.io_cmds_active: 896 dev.mps.0.chain_free: 39 dev.mps.0.io_cmds_active: 12 root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active dev.mps.3.chain_free: 412 dev.mps.3.io_cmds_active: 572 dev.mps.2.chain_free: 718 dev.mps.2.io_cmds_active: 687 dev.mps.1.chain_free: 211 dev.mps.1.io_cmds_active: 906 dev.mps.0.chain_free: 65 dev.mps.0.io_cmds_active: 144 root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active dev.mps.3.chain_free: 500 dev.mps.3.io_cmds_active: 629 dev.mps.2.chain_free: 623 dev.mps.2.io_cmds_active: 676 dev.mps.1.chain_free: 251 dev.mps.1.io_cmds_active: 907 dev.mps.0.chain_free: 139 dev.mps.0.io_cmds_active: 144 [...] root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active dev.mps.3.chain_free: 1874 dev.mps.3.io_cmds_active: 78 dev.mps.2.chain_free: 1888 dev.mps.2.io_cmds_active: 64 dev.mps.1.chain_free: 1922 dev.mps.1.io_cmds_active: 42 dev.mps.0.chain_free: 1936 dev.mps.0.io_cmds_active: 48 root@edge02:/ # sysctl dev.mps | grep -e chain_free: -e io_cmds_active dev.mps.3.chain_free: 1890 dev.mps.3.io_cmds_active: 78 dev.mps.2.chain_free: 1890 dev.mps.2.io_cmds_active: 82 dev.mps.1.chain_free: 1729 dev.mps.1.io_cmds_active: 150 dev.mps.0.chain_free: 1893 dev.mps.0.io_cmds_active: 57 What this mean? Why with allocated 2048 chains per controller I see 65 free and 144 allocated? How I got 976 active commands? I am use SATA HDD suported only 32 tags on NCQ, with 8 ports this maximum 256 outstanding commands per controller. How I can resolve this issue?