From owner-freebsd-current@FreeBSD.ORG Wed Jun 22 14:47:20 2005 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 96C2B16A41C for ; Wed, 22 Jun 2005 14:47:20 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4571C43D1F for ; Wed, 22 Jun 2005 14:47:20 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id A45CB46B08; Wed, 22 Jun 2005 10:47:19 -0400 (EDT) Date: Wed, 22 Jun 2005 15:50:24 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Emanuel Strobl In-Reply-To: <200506221606.42862@harrymail> Message-ID: <20050622154538.H26664@fledge.watson.org> References: <200506091423.39940@harrymail> <200506221554.41750@harrymail> <42B96E66.4040503@samsco.org> <200506221606.42862@harrymail> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= , freebsd-current@freebsd.org Subject: Re: lapic@2k interrukts eating CPU cycles X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jun 2005 14:47:20 -0000 On Wed, 22 Jun 2005, Emanuel Strobl wrote: >> Because you have 'options APIC' in your kernel, and your running a post >> 5.2 version of FreeBSD that has a massively improved interrupt routing >> system in it. > > I've been running 5-current on that machine with "options APIC" for a > long time, back to when I had to include NO-MIXEDMODE to get it working > until I think Robeert Watson gave me some patches. But I never had lapic > so I was wondering... Might have been John Baldwin. In 5.x, timer interrupts are generated by a general purpose programmable timer and delivered as interrupts to one of your CPUs on an MP system, or your only CPU on a UP system. Depending on your hardware, where the timer went varied a bit -- typically, round robin, or maybe your first CPU or third CPU. The timer tick would then be broadcasted to the other CPUs using an inter-processor interrupt (IPI), so that all CPUs would see it for the purposes of timer-based preemption, profiling, statistics, and so on. In 6.x, we use the timer in the CPU-local apic (LAPIC) to interrupt each CPU, rather than delivering all the interrupts to a particular CPU and forwarding them. This lowers the IPI traffic overhead, which can be substantial due to synchronizing and preventing IPI deadlock. In particular, spin locks are used around IPI interactions to prevent two or more CPUs from deadlocking by sending each other IPIs simultaneously, and therefore being unable to acknowledge each other (there's a whole literature on why and how to do this, if you google or check a decent MP OS text). So in 5.x, you saw the primary timer interrupt traffic explicitly, and most interrupt monitoring tools didn't monitor IPI load. In 6.x, you see the timer interrupt traffic on each CPU as lapic load for the CPU. In 6.x, we currently don't monitor IPI interrupt load, but if we did in 5.x and 6.x, you'd see that the IPI traffic level is much lower, making up for the increased LAPIC interrupt traffic. I have a feature request in to John to add statistics gathering on IPIs, since he's currently reworking the interrupt paths. Robert N M Watson