From owner-freebsd-hackers@freebsd.org Fri Mar 18 18:16:28 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9CD1AD5232 for ; Fri, 18 Mar 2016 18:16:28 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from mx0.deglitch.com (mx0.deglitch.com [IPv6:2a00:13c0:63:7194:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id A2AEC1D52; Fri, 18 Mar 2016 18:16:28 +0000 (UTC) (envelope-from stas@FreeBSD.org) Received: from [IPv6:2620:10d:c082:1803:9c65:ebd8:7cd2:7e46] (unknown [IPv6:2620:10d:c090:200::f:9a2b]) by mx0.deglitch.com (Postfix) with ESMTPSA id 1125D8FC0B; Fri, 18 Mar 2016 11:16:23 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: Peformance issues with r278325 From: Stanislav Sedov In-Reply-To: <3277812.DVsZx4uMun@ralph.baldwin.cx> Date: Fri, 18 Mar 2016 11:16:21 -0700 Cc: Adrian Chadd , Konstantin Belousov , "freebsd-hackers@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <8EE51E0E-41F4-4B5A-A755-B58E8E1D1776@FreeBSD.org> References: <8403291.NqUNo0Qq5W@ralph.baldwin.cx> <3277812.DVsZx4uMun@ralph.baldwin.cx> To: John Baldwin X-Mailer: Apple Mail (2.3112) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Mar 2016 18:16:29 -0000 > On Mar 18, 2016, at 10:37 AM, John Baldwin wrote: >=20 > On Thursday, March 17, 2016 10:45:39 AM Adrian Chadd wrote: >> On 17 March 2016 at 10:23, John Baldwin wrote: >>=20 >>>=20 >>> I had not expected this commit to have this impact, but Konstantin = is correct. >>> I wonder if simply reducing the DELAY() from 5 down to 1 or so would = be >>> sufficient? (Note that you need to adjust the prior loop to use +=3D = 1 instead >>> of +=3D 5 in that case.) >>>=20 >>> Note that DETECT_DEADLOCK is not enabled by default, so the = AFTER_SPIN case >>> (which waits for an IPI just sent to be delivered) shouldn't be = enabled (and >>> in fact I'd like to just remove that code entirely). This means = that only >>> BEFORE_SPIN should be spinning, and it should only be spinning if a = CPU sends >>> IPIs back to back such that the previous IPI is still pending (not = yet >>> delivered) when the CPU wants to send another IPI. >>>=20 >>> We can probably assume a TSC if we have SMP, so if changing the = delay from 5 >>> to 1 doesn't work we can try just using the TSC directly to control = the >>> spin length and go back to using a simple pause. >>>=20 >>> I have an old set of changes that might also be interesting that = permit >>> TLB shootdown IPI handlers to run while spinlocks are held (by using = cr8/TPR >>> to control interrupts when a spinlock is held instead of disabling = all >>> interrupts). I haven't found a workload where that helped yet. = However, >>> yours might be an interesting workload to try those changes out on. >>=20 >> Do you think it's worth just reverting it for now just so it lands in >> 10.3-RELEASE? >=20 > Probably. If the '1' change fixes it that is a simple test, otherwise = we > can revert in 10.x. I think I'll likely just convert it to use a = direct > TSC delay loop always in HEAD (assuming that verifies ok in testing as = well). >=20 FWIW we are currently testing the delay '1' change. Unfortunately, the = test is not easy to repeat (we didn't find a synthetic one yet that results in = the same outcome), so it does take more time that I would like. Will follow up = with the=20 results. We did try HEAD as well a while ago, and although it exhibited the same = pattern. However it did not utilize the x2apic unfortunately, as it does seem to = be disables in the BIOS (FreeBSD reports it being disables in the DMAR table). Thanks for looking into it! -- Stanislav Sedov ST4096-RIPE