Date: Fri, 18 Mar 2016 14:05:39 -0700 From: John Baldwin <jhb@freebsd.org> To: "K. Macy" <kmacy@freebsd.org> Cc: Stanislav Sedov <stas@freebsd.org>, Adrian Chadd <adrian.chadd@gmail.com>, Konstantin Belousov <kostikbel@gmail.com>, Ryan Stone <rysto32@gmail.com>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Peformance issues with r278325 Message-ID: <1938968.vRTG1GJntD@ralph.baldwin.cx> In-Reply-To: <CAHM0Q_MaHwJhd2YWPV3V8R8KGkM=GwPh79jyEcsaro%2B_d__e9A@mail.gmail.com> References: <FA50A68E-7F3D-4361-8A8A-EB7F97EF3D00@FreeBSD.org> <3213721.SRyKE0LtiG@ralph.baldwin.cx> <CAHM0Q_MaHwJhd2YWPV3V8R8KGkM=GwPh79jyEcsaro%2B_d__e9A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Friday, March 18, 2016 01:11:15 PM K. Macy wrote:
> On Friday, March 18, 2016, John Baldwin <jhb@freebsd.org> wrote:
>
> > On Friday, March 18, 2016 11:58:19 AM Stanislav Sedov wrote:
> > >
> > > > On Mar 18, 2016, at 11:49 AM, Ryan Stone <rysto32@gmail.com
> > <javascript:;>> wrote:
> > > >
> > > > On Fri, Mar 18, 2016 at 1:37 PM, John Baldwin <jhb@freebsd.org
> > <javascript:;>> wrote:
> > > > I think I'll likely just convert it to use a direct
> > > > TSC delay loop always in HEAD (assuming that verifies ok in testing as
> > well).
> > > >
> > > > Couldn't that work incorrectly on VM guests? The tsc is not
> > guaranteed to be monotonic in that environment.
> > > >
> > >
> > > Another concern is SMP systems where the clock is not synchronized. SMP
> > TSC requires
> > > a complicated setup procedure on the system boot which is not followed
> > properly by all
> > > vendors, and I have seen some recent systems with SMP TSC skew.
> > >
> > > I'm afraid that using TSC in this code will make FreeBSD unusable on
> > such (arguably buggy)
> > > systems.
> >
> > Eh, SMP does not matter here. DELAY() already uses TSC on FreeBSD. The
> > current thread is pinned to a single CPU in lapic_ipi_wait(). The idea
> > would be to do this:
> >
> > deadline = rdtsc() + freq * delay / 1000000;
> >
> > while (rdtsc() < deadline) {
> > if (APIC_DELSTAT_IDLE)
> > return (1);
> > ia32_pause();
> > }
> >
> >
> >
> In a VM the thread is pinned to the _vcpu_. You have no control over what
> physical core the underlying thread is running on. If the vcpu is migrated
> to a different package there is a possibility of the next TSC reading be a
> lower value. The rdtsc instruction reads the value of the physical core.
> Although I doubt this is a problem in practice, it is a possibility if
> vcpus are not pinned for the lifetime of the VM.
Again, DELAY() already makes this assumption on FreeBSD/x86. If this is
broken, it's broken in far, far many more places than this. In addition, a
hypervisor is capable of handling this if it uses the TSC adjustment field
in the VMCS. OTOH, TSC's on modern systems generally are in sync. Invariant
TSCs are generated from a timer in the uncore on all the i[357] CPUs AFAIK, so
they are identical across cores and threads within a package. They are also
in sync across packages on all of the i[357] systems I have worked with (and
generations prior to those) in the last 7-8 years (barring one machine I'm
aware of that was a quad-socket i7 box where the BIOS overwrote the TSC of
the first core in each package on each reboot to try to reset it to zero on
each boot).
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1938968.vRTG1GJntD>