From owner-freebsd-hackers@freebsd.org Sat Mar 19 08:08:42 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2FA74AD6F46 for ; Sat, 19 Mar 2016 08:08:42 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9D3AF0D; Sat, 19 Mar 2016 08:08:41 +0000 (UTC) (envelope-from gljennjohn@gmail.com) Received: by mail-wm0-x241.google.com with SMTP id l68so10562462wml.3; Sat, 19 Mar 2016 01:08:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; bh=pzEu347SieYbhlSwmPfTonQWbm/IB8lxI7TyLWmFgXg=; b=L/+MXmG3ykKbWwM8Fx6IWPZlEpYtKHZCzGCHTANVpl1w1FCWL7V/gqR2CjpTSQF3yO clasPu85m1s2UBsjsnaDzctMGdTETL2EepOqFzTNY5W364a4HNQXaWZUogpHtufO/Vgq 0du++dVXbLyx5VJTSMjg7rcZfFtwbepa8UspvUUg6MD5tFpC4dDFoS+3TnC3Md4EzS7K 2Zi0A+Js/HxMDOStgQrB0f0SXkX/aQga1KiFesW1t12XpV5pzRMBJUuq0wIJ3h7p2k1Q b+l6Ipt/05CiYL8NLj4vjy4z9Hi9MLUPiXsUAhjesF7eEHbz8uK7EVITPNYOq8CZS8K+ /I5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:reply-to:mime-version:content-transfer-encoding; bh=pzEu347SieYbhlSwmPfTonQWbm/IB8lxI7TyLWmFgXg=; b=RA3MSc88uILgkRcluGkWt8yb9UZtyKuMg41SQIJurjtDl41lG34CfyrkaqTpDAOgCj cVU5o79Md3Vq1CPHCmutjF56NYWW0ty3lRNYKVoUk/4s+h+mm0048TV+mQWaftQdypMe IKa8X55qJkuS6leIdn+w+5dPvdV0gmoSbskdJ3zLV/9Uhrt9TDsK3PWMMM6YYCGnXzFg EinS9P3xMFltWlbw5BWJx6cKkU1XCl6U1V3ZdUW0vKQE7l6UtX9qZmkxiGAV7Uz4CNhG 1VYIfhOAAoovYc78hFTpNfRCtEd/wBnVqvFqHKsCKKSjfGL2naxupP/N+mFFbcl/uig5 vz7Q== X-Gm-Message-State: AD7BkJIpRumkWpRJkIdt7MJTzCyNxWiY8txgYSUPFn3T1zEt9SV/jDcnebW6l5GLDQQjCw== X-Received: by 10.28.104.131 with SMTP id d125mr2892664wmc.99.1458374920334; Sat, 19 Mar 2016 01:08:40 -0700 (PDT) Received: from ernst.home (p4FCA7056.dip0.t-ipconnect.de. [79.202.112.86]) by smtp.gmail.com with ESMTPSA id 82sm2574421wmd.4.2016.03.19.01.08.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 Mar 2016 01:08:39 -0700 (PDT) Date: Sat, 19 Mar 2016 09:08:37 +0100 From: Gary Jennejohn To: John Baldwin Cc: "K. Macy" , Stanislav Sedov , Adrian Chadd , Konstantin Belousov , Ryan Stone , "freebsd-hackers@freebsd.org" Subject: Re: Peformance issues with r278325 Message-ID: <20160319090837.7799561c@ernst.home> In-Reply-To: <1938968.vRTG1GJntD@ralph.baldwin.cx> References: <3213721.SRyKE0LtiG@ralph.baldwin.cx> <1938968.vRTG1GJntD@ralph.baldwin.cx> Reply-To: gljennjohn@gmail.com X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Mar 2016 08:08:42 -0000 On Fri, 18 Mar 2016 14:05:39 -0700 John Baldwin wrote: > On Friday, March 18, 2016 01:11:15 PM K. Macy wrote: > > On Friday, March 18, 2016, John Baldwin wrote: > > > > > On Friday, March 18, 2016 11:58:19 AM Stanislav Sedov wrote: > > > > > > > > > On Mar 18, 2016, at 11:49 AM, Ryan Stone > > > wrote: > > > > > > > > > > On Fri, Mar 18, 2016 at 1:37 PM, John Baldwin > > > wrote: > > > > > I think I'll likely just convert it to use a direct > > > > > TSC delay loop always in HEAD (assuming that verifies ok in testing as > > > well). > > > > > > > > > > Couldn't that work incorrectly on VM guests? The tsc is not > > > guaranteed to be monotonic in that environment. > > > > > > > > > > > > > Another concern is SMP systems where the clock is not synchronized. SMP > > > TSC requires > > > > a complicated setup procedure on the system boot which is not followed > > > properly by all > > > > vendors, and I have seen some recent systems with SMP TSC skew. > > > > > > > > I'm afraid that using TSC in this code will make FreeBSD unusable on > > > such (arguably buggy) > > > > systems. > > > > > > Eh, SMP does not matter here. DELAY() already uses TSC on FreeBSD. The > > > current thread is pinned to a single CPU in lapic_ipi_wait(). The idea > > > would be to do this: > > > > > > deadline = rdtsc() + freq * delay / 1000000; > > > > > > while (rdtsc() < deadline) { > > > if (APIC_DELSTAT_IDLE) > > > return (1); > > > ia32_pause(); > > > } > > > > > > > > > > > In a VM the thread is pinned to the _vcpu_. You have no control over what > > physical core the underlying thread is running on. If the vcpu is migrated > > to a different package there is a possibility of the next TSC reading be a > > lower value. The rdtsc instruction reads the value of the physical core. > > Although I doubt this is a problem in practice, it is a possibility if > > vcpus are not pinned for the lifetime of the VM. > > Again, DELAY() already makes this assumption on FreeBSD/x86. If this is > broken, it's broken in far, far many more places than this. In addition, a > hypervisor is capable of handling this if it uses the TSC adjustment field > in the VMCS. OTOH, TSC's on modern systems generally are in sync. Invariant > TSCs are generated from a timer in the uncore on all the i[357] CPUs AFAIK, so > they are identical across cores and threads within a package. They are also > in sync across packages on all of the i[357] systems I have worked with (and > generations prior to those) in the last 7-8 years (barring one machine I'm > aware of that was a quad-socket i7 box where the BIOS overwrote the TSC of > the first core in each package on each reboot to try to reset it to zero on > each boot). > Just out of curiosity, can one assume that this is also true for AMD CPUs? I'm using a Phenom II X6 1090T in my box. -- Gary Jennejohn