From owner-freebsd-drivers@freebsd.org Tue Mar 27 10:26:12 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CC11F6AFFA for ; Tue, 27 Mar 2018 10:26:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1A177C321 for ; Tue, 27 Mar 2018 10:26:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w2RAQ0Co030191 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 27 Mar 2018 13:26:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w2RAQ0Co030191 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w2RAQ0qZ030189; Tue, 27 Mar 2018 13:26:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 27 Mar 2018 13:26:00 +0300 From: Konstantin Belousov To: "Kononov, Oleksandr" Cc: "freebsd-drivers@freebsd.org" , "Vanco, Juraj" Subject: Re: FreeBSD 11.1 contigfree performance issue Message-ID: <20180327102600.GY76926@kib.kiev.ua> References: <865AA1660A1A014C99D99B800FA40800813681@IRSMSX101.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <865AA1660A1A014C99D99B800FA40800813681@IRSMSX101.ger.corp.intel.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2018 10:26:12 -0000 On Tue, Mar 27, 2018 at 09:53:11AM +0000, Kononov, Oleksandr wrote: > > I am using FreeBSD 11.1 -RELEASE-amd64 running on a single 32 core CPU and am having issues with contigmalloc performance. > Timing the function using rdtsc shows that it uses up on average about 10 million cycles on that function along. > Using the same code, FreeBSD and timing method I ran it on anther machine on two CPU's with a total of 32 cores. > This gave about 12 thousand cycles on that function. > > Digging through the source code (on the single CPU) I found that > > smp_targeted_tlb_shootdown function in /usr/src/sys/x86/x86/mp_x86.c > cause the majority of performance hit due to some cores remaining in a paused state longer after the > interrupt was send to them. > > > I attached a sample code and Makefile in this email. > > Steps to recreate (and show rdtsc cycles): > > $ make > $ kldload ./test.ko > $ dmesg > > If anyone has any idea what is the cause of this issue, it would be greatly appreciated. You noted that CPUs have very long time coming out of the idle state. What is the output of sysctl dev.cpu ? In particular, look at the cx_* MIBs and if you have configured deep sleep modes, try to step it back. Also look at what idle method (cx_method) is used, and if MWAIT is problematic, change to the legacy idling, see acpi(4) and debug.acpi.disable="mwait" knob. What are those CPUs ? Wouldn't such high latency of waking up cores make the interactive performance of the machine a mess already. In principle, we can avoid sending shutdowns to the idle cores, instead doing global TLB flush on wakeup of the core if some percpu flag is set. But as I noted in the paragraph before, it should be quite bad even with IPI latency fixed, for other reasons.