Date: Sat, 26 Nov 2016 14:37:45 -0800 From: Jason Harmening <jason.harmening@gmail.com> To: Konstantin Belousov <kostikbel@gmail.com> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: huge nanosleep variance on 11-stable Message-ID: <CAM=8qanhbJ89CiA%2B7vUxKEX6YjMH0uwJ25Qng5y2YW7oE6SgRg@mail.gmail.com> In-Reply-To: <CAM=8qakFsacJcURw6SN5LksbOYAqwgqu0BV6Kym_0x30DxF0Dg@mail.gmail.com> References: <c88341e2-4c52-ed3c-a469-6446da4415f4@gmail.com> <6167392c-c37a-6e39-aa22-ca45435d6088@gmail.com> <20161102075509.GF54029@kib.kiev.ua> <3620f62e-0f4c-2d62-dcf8-e2fdff459250@gmail.com> <20161102162808.GI54029@kib.kiev.ua> <20161125092503.GZ54029@kib.kiev.ua> <CAM=8qakFsacJcURw6SN5LksbOYAqwgqu0BV6Kym_0x30DxF0Dg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I can confirm this patch works. HPET is now chosen over LAPIC as the
eventtimer source, and the system works smoothly without disabling C2 or
mwait.
On Fri, Nov 25, 2016 at 4:12 AM, Jason Harmening <jason.harmening@gmail.com>
wrote:
> On Fri, Nov 25, 2016 at 1:25 AM, Konstantin Belousov <kostikbel@gmail.com>
> wrote:
>
>> On Wed, Nov 02, 2016 at 06:28:08PM +0200, Konstantin Belousov wrote:
>> > On Wed, Nov 02, 2016 at 09:18:15AM -0700, Jason Harmening wrote:
>> > > I think you are probably right. Hacking out the Intel-specific
>> > > additions to C-state parsing in acpi_cpu_cx_cst() from r282678 (thus
>> > > going back to sti;hlt instead of monitor+mwait at C1) fixed the
>> problem
>> > > for me. But r282678 also had the effect of enabling C2 and C3 on my
>> > > system, because ACPI only presents MWAIT entries for those states and
>> > > not p_lvlx.
>> > You can do the same with "debug.acpi.disabled=mwait" loader tunable
>> > without hacking the code. And set sysctl hw.acpi.cpu.cx_lowest to C1 to
>> > enforce use of hlt instruction even when mwait states were requested.
>>
>> I believe I now understood the problem. First, I got the definitive
>> confirmation that LAPIC timer on Nehalems is stopped in any C mode
>> higher than C1/C1E, i.e. even if C2 is enabled LAPIC eventtimer cannot
>> be used. This is consistent with the ARAT CPUID bit CPUID[0x6].eax[2]
>> reported zero.
>>
>> On SandyBridge and IvyBridge CPUs, it seems that ARAT might be both 0
>> and 1 according to the same source, but all CPUs I saw have ARAT = 1.
>> And for Haswell and later generations, ARAT is claimed to be always
>> implemented.
>>
>> The actual issue is somewhat silly bug, I must admit: if ncpus >= 8, and
>> non-FSB interrupt routing from HPET, default HPET eventtimer quality 450
>> is reduced by 100, i.e. it is 350. OTOH, LAPIC default quality is 600
>> and it is reduced by 200 if ARAT is not reported. We end up with HPET
>> quality 350 < LAPIC quality 400, despite ARAT is not set.
>>
>> The patch below sets LAPIC eventtimer quality to 100 if not ARAT. Also
>> I realized that there is no reason to disable deadline mode regardless
>> of ARAT.
>>
>> diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
>> index d9a3453..1b1547d 100644
>> --- a/sys/x86/x86/local_apic.c
>> +++ b/sys/x86/x86/local_apic.c
>> @@ -478,8 +478,9 @@ native_lapic_init(vm_paddr_t addr)
>> lapic_et.et_quality = 600;
>> if (!arat) {
>> lapic_et.et_flags |= ET_FLAGS_C3STOP;
>> - lapic_et.et_quality -= 200;
>> - } else if ((cpu_feature & CPUID_TSC) != 0 &&
>> + lapic_et.et_quality = 100;
>> + }
>> + if ((cpu_feature & CPUID_TSC) != 0 &&
>> (cpu_feature2 & CPUID2_TSCDLT) != 0 &&
>> tsc_is_invariant && tsc_freq != 0) {
>> lapic_timer_tsc_deadline = 1;
>>
>> Ah, that makes sense. Thanks!
>
> I'll try the patch as soon as I get back from vacation. I've been able to
> verify that setting cx_lowest and disabling mwait fixes the problem without
> hacking the code. But I've been too busy at $(WORK) to check anything
> else, namely whether forcing HPET would also fix the problem.
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM=8qanhbJ89CiA%2B7vUxKEX6YjMH0uwJ25Qng5y2YW7oE6SgRg>