Date: Mon, 06 Feb 2006 14:00:34 +0100 From: Andre Oppermann <andre@freebsd.org> To: Andrew Gallatin <gallatin@cs.duke.edu> Cc: freebsd-current@freebsd.org Subject: Re: machdep.cpu_idle_hlt and SMP perf? Message-ID: <43E74872.7000002@freebsd.org> In-Reply-To: <17379.56708.421007.613310@grasshopper.cs.duke.edu> References: <17379.56708.421007.613310@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Andrew Gallatin wrote: > Why dooes machdep.cpu_idle_hlt=1 drop my 10GbE network rx > performance by a considerable amount (7.5Gbs -> 5.5Gbs)? > > I've (blindly) tried leaving machdep.cpu_idle_hlt=1 enabled > and playing with the vast array of kern.sched.ipiwakeup.* sysctls, > but receive performance remains limited to ~5.5Gb/sec or less. > > This is an 'AMD Athlon(tm) 64 X2 Dual Core Processor 3800+' running > FreeBSD-current as of about one week ago. The interrupt load is > about 22,000 device interrupts/sec (ithreaded). Interestingly, > the more I decrease the interrupt load by increasing the interrupt > coalescing timer, the worse the machdep.cpu_idle_hlt=1 case does. > > Is this just a case of the wakeup IPI taking a long time or blocking > on some lock? This may be the same problem OpenBSD has fixed last year in the handling of the idle loop. From the kerneltrap posting: % "In the following article, Bob provides a first-person account of tracking % down what began simply as a RAID performance issue, but ultimately turned % out to be a problem with the idle loop that when fixed resulted in an % impressive performance boost. Bob noted, "the idle loop is where the kernel % spins when there is no work to do in userland, because of this, it's also % where we catch and service many of our interrupts from drivers that may queue % work to the device and then tsleep waiting for an interrupt from the card % saying the work is done." Bob went on to explain that prior to today's fix, % interrupts were handled appropriately when there was userland work happening, % but not when there was nothing happening in userland and the kernel was simply % waiting for device input/output. Read on for Bob's full account of the day, % leading up to the discovery of the problem and the implementation of the fix, % including performance numbers." Here is the more specific information: http://kerneltrap.org/node/5169 First commit message: http://marc.theaimsgroup.com/?l=openbsd-cvs&m=111692513727274&w=2 The MFC with all changes in one commit message: http://marc.theaimsgroup.com/?l=openbsd-cvs&m=111859519015510&w=2 http://www.openbsd.org/cgi-bin/cvsweb/src/sys/arch/i386/i386/locore.s http://www.openbsd.org/cgi-bin/cvsweb/src/sys/arch/i386/i386/apm.c -- Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43E74872.7000002>