From owner-freebsd-hackers@freebsd.org Fri Mar 18 18:05:15 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2A24AD4D59 for ; Fri, 18 Mar 2016 18:05:15 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 939181053; Fri, 18 Mar 2016 18:05:15 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A3C63B94B; Fri, 18 Mar 2016 14:05:14 -0400 (EDT) From: John Baldwin To: Adrian Chadd Cc: Konstantin Belousov , Stanislav Sedov , "freebsd-hackers@freebsd.org" Subject: Re: Peformance issues with r278325 Date: Fri, 18 Mar 2016 10:37:55 -0700 Message-ID: <3277812.DVsZx4uMun@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: References: <8403291.NqUNo0Qq5W@ralph.baldwin.cx> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 18 Mar 2016 14:05:14 -0400 (EDT) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Mar 2016 18:05:15 -0000 On Thursday, March 17, 2016 10:45:39 AM Adrian Chadd wrote: > On 17 March 2016 at 10:23, John Baldwin wrote: > > > > > I had not expected this commit to have this impact, but Konstantin is correct. > > I wonder if simply reducing the DELAY() from 5 down to 1 or so would be > > sufficient? (Note that you need to adjust the prior loop to use += 1 instead > > of += 5 in that case.) > > > > Note that DETECT_DEADLOCK is not enabled by default, so the AFTER_SPIN case > > (which waits for an IPI just sent to be delivered) shouldn't be enabled (and > > in fact I'd like to just remove that code entirely). This means that only > > BEFORE_SPIN should be spinning, and it should only be spinning if a CPU sends > > IPIs back to back such that the previous IPI is still pending (not yet > > delivered) when the CPU wants to send another IPI. > > > > We can probably assume a TSC if we have SMP, so if changing the delay from 5 > > to 1 doesn't work we can try just using the TSC directly to control the > > spin length and go back to using a simple pause. > > > > I have an old set of changes that might also be interesting that permit > > TLB shootdown IPI handlers to run while spinlocks are held (by using cr8/TPR > > to control interrupts when a spinlock is held instead of disabling all > > interrupts). I haven't found a workload where that helped yet. However, > > yours might be an interesting workload to try those changes out on. > > Do you think it's worth just reverting it for now just so it lands in > 10.3-RELEASE? Probably. If the '1' change fixes it that is a simple test, otherwise we can revert in 10.x. I think I'll likely just convert it to use a direct TSC delay loop always in HEAD (assuming that verifies ok in testing as well). -- John Baldwin