From owner-freebsd-hackers@freebsd.org Thu Mar 17 17:45:40 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20CB8AD30BF for ; Thu, 17 Mar 2016 17:45:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22d.google.com (mail-io0-x22d.google.com [IPv6:2607:f8b0:4001:c06::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E11CE981; Thu, 17 Mar 2016 17:45:39 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x22d.google.com with SMTP id g184so51094295ioa.3; Thu, 17 Mar 2016 10:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=UCuHFb65NPOBHih+9SxyFiiBOSlc4sL2tNkDZwQS3oE=; b=EeGGEun4uYUFUU5+I7kapBCZ6GH1QqIrzY2Z94dRBRJkqL8YZAgOzqR+rx9/in7XNx g3c6zJ8hxUmZ0MgGw7len6YamoSKn8oAhOSRBEx2HlRF9WDAsZHCOMaWHx71E41Q/adY ADExLkh+o8Rad0OhKprpYQ70q10O/m80COsQLb6Nfd5ZeyWjCM4GhC4KTorQQiz2/Umm pNxc0WPRrmLMafh/F0ICpKsBal6nBga6/HIMqZ2J/D13GZV+wR/I2HwEhFCfhXFYLgj/ D58bayjFPbD3mzB1mTkiT8JCIF8+sPjXqmT53rFA1foBiqVkJpEkF/Q6gkdmvDHZIDqs j7Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=UCuHFb65NPOBHih+9SxyFiiBOSlc4sL2tNkDZwQS3oE=; b=dr9c8ucoTJjG/6gUW5ZicFWl7BZEWBdrFRnCabVgD13TLZyocKUD24NMfS1z8o00o0 KKtF+wSm8P2FWkDYuloYuQXCE80ljruN0+tboY2ZiUfMJqnkp0SJb2whBi8H4Vh4xToi DCwY5Y7uEljiHvbPqR0A1d8J9ActU+eE/sqF+VEOZ068Ww6r5xMKSzcwamg+yCyfwzpF b0wEbmivMuYfKkKmm+GfIJjBEwEVyZYIixs9kYeWwsA1QiKOTASYYLDOmDcTCvm8Ze3t Z6w3UW3lBua3VO7JhtGTkgbCKX+Dn7FZQ+l3qHGp8m3gIXUIletMZ8jTFAbTCpas39K1 IPsA== X-Gm-Message-State: AD7BkJIcnMq2Z/iuIp+3fDoF8GEpJrH0e5LbCPmRq9uz5MEUY/76/UYgMTUEiBuJSvmRe81nplNMNb7HtxtelA== MIME-Version: 1.0 X-Received: by 10.107.11.162 with SMTP id 34mr10728098iol.165.1458236739369; Thu, 17 Mar 2016 10:45:39 -0700 (PDT) Received: by 10.36.14.19 with HTTP; Thu, 17 Mar 2016 10:45:39 -0700 (PDT) In-Reply-To: <8403291.NqUNo0Qq5W@ralph.baldwin.cx> References: <20160317053704.GP1741@kib.kiev.ua> <8403291.NqUNo0Qq5W@ralph.baldwin.cx> Date: Thu, 17 Mar 2016 10:45:39 -0700 Message-ID: Subject: Re: Peformance issues with r278325 From: Adrian Chadd To: John Baldwin Cc: Konstantin Belousov , Stanislav Sedov , "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Mailman-Approved-At: Thu, 17 Mar 2016 18:31:10 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Mar 2016 17:45:40 -0000 On 17 March 2016 at 10:23, John Baldwin wrote: > > I had not expected this commit to have this impact, but Konstantin is correct. > I wonder if simply reducing the DELAY() from 5 down to 1 or so would be > sufficient? (Note that you need to adjust the prior loop to use += 1 instead > of += 5 in that case.) > > Note that DETECT_DEADLOCK is not enabled by default, so the AFTER_SPIN case > (which waits for an IPI just sent to be delivered) shouldn't be enabled (and > in fact I'd like to just remove that code entirely). This means that only > BEFORE_SPIN should be spinning, and it should only be spinning if a CPU sends > IPIs back to back such that the previous IPI is still pending (not yet > delivered) when the CPU wants to send another IPI. > > We can probably assume a TSC if we have SMP, so if changing the delay from 5 > to 1 doesn't work we can try just using the TSC directly to control the > spin length and go back to using a simple pause. > > I have an old set of changes that might also be interesting that permit > TLB shootdown IPI handlers to run while spinlocks are held (by using cr8/TPR > to control interrupts when a spinlock is held instead of disabling all > interrupts). I haven't found a workload where that helped yet. However, > yours might be an interesting workload to try those changes out on. Do you think it's worth just reverting it for now just so it lands in 10.3-RELEASE? -adrian