From owner-cvs-src@FreeBSD.ORG Tue Feb 28 22:57:33 2006 Return-Path: X-Original-To: cvs-src@freebsd.org Delivered-To: cvs-src@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CE50416A420; Tue, 28 Feb 2006 22:57:33 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (66-23-211-162.clients.speedfactory.net [66.23.211.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id A1B1E43D6E; Tue, 28 Feb 2006 22:57:23 +0000 (GMT) (envelope-from jhb@freebsd.org) Received: from localhost (john@localhost [127.0.0.1]) by server.baldwin.cx (8.13.4/8.13.4) with ESMTP id k1SMvLVr020347; Tue, 28 Feb 2006 17:57:21 -0500 (EST) (envelope-from jhb@freebsd.org) From: John Baldwin To: Scott Long Date: Tue, 28 Feb 2006 17:58:24 -0500 User-Agent: KMail/1.9.1 References: <200602282224.k1SMOtJt070241@repoman.freebsd.org> <200602281735.12240.jhb@freebsd.org> <4404D37E.9040502@samsco.org> In-Reply-To: <4404D37E.9040502@samsco.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200602281758.26808.jhb@freebsd.org> X-Virus-Scanned: ClamAV 0.87.1/1306/Tue Feb 28 04:50:04 2006 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-1.4 required=4.2 tests=ALL_TRUSTED autolearn=ham version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on server.baldwin.cx Cc: cvs-src@freebsd.org, src-committers@freebsd.org, cvs-all@freebsd.org Subject: Re: cvs commit: src/sys/amd64/amd64 intr_machdep.c io_apic.c local_apic.c mp_machdep.c src/sys/amd64/include apicvar.h intr_machdep.h src/sys/amd64/isa atpic.c src/sys/i386/i386 intr_machdep.c io_apic.c local_apic.c mp_machdep.c ... X-BeenThere: cvs-src@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CVS commit messages for the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2006 22:57:34 -0000 On Tuesday 28 February 2006 17:49, Scott Long wrote: > John Baldwin wrote: > > On Tuesday 28 February 2006 17:24, John Baldwin wrote: > > > >>jhb 2006-02-28 22:24:55 UTC > >> > >> FreeBSD src repository > >> > >> Modified files: > >> sys/amd64/amd64 intr_machdep.c io_apic.c local_apic.c > >> mp_machdep.c > >> sys/amd64/include apicvar.h intr_machdep.h > >> sys/amd64/isa atpic.c > >> sys/i386/i386 intr_machdep.c io_apic.c local_apic.c > >> mp_machdep.c > >> sys/i386/include apicvar.h intr_machdep.h > >> sys/i386/isa atpic.c > >> Log: > >> Rework how we wire up interrupt sources to CPUs: > >> - Throw out all of the logical APIC ID stuff. The Intel docs are somewhat > >> ambiguous, but it seems that the "flat" cluster model we are currently > >> using is only supported on Pentium and P6 family CPUs. The other > >> "hierarchy" cluster model that is supported on all Intel CPUs with > >> local APICs is severely underdocumented. For example, it's not clear > >> if the OS needs to glean the topology of the APIC hierarchy from > >> somewhere (neither ACPI nor MP Table include it) and setup the logical > >> clusters based on the physical hierarchy or not. Not only that, but on > >> certain Intel chipsets, even though there were 4 CPUs in a logical > >> cluster, all the interrupts were only sent to one CPU anyway. > >> - We now bind interrupts to individual CPUs using physical addressing via > >> the local APIC IDs. This code has also moved out of the ioapic PIC > >> driver and into the common interrupt source code so that it can be > >> shared with MSI interrupt sources since MSI is addressed to APICs the > >> same way that I/O APIC pins are. > > > > - Use fixed delivery mode rather than low priority, as apparently low > > priority mode only works with logical APIC IDs (though this is not > > clearly documented that I've seen). Also, I've observed behavior where > > low priority mode will deliver interrupts to a different CPU than the > > one you've specifically routed the IRQ to using physical addressing > > on certain Intel chipsets. FYI, we use low priority delivery method > > with a wildcard physical address on all released versions of FreeBSD, > > back to revision 1.1 of sys/i386/i386/mpapic.c. > > > >> - Interrupt source classes grow a new method pic_assign_cpu() to bind an > >> interrupt source to a specific local APIC ID. > >> - The SMP code now tells the interrupt code which CPUs are avaiable to > >> handle interrupts in a simpler and more intuitive manner. For one thing, > >> it means we could now choose to not route interrupts to HT cores if we > >> wanted to (this code is currently in place in fact, but under an #if 0 > >> for now). > >> - For now we simply do static round-robin of IRQs to CPUs when the first > >> interrupt handler just as before, with the change that IRQs are now > >> bound to individual CPUs rather than groups of up to 4 CPUs. > >> - Because the IRQ to CPU mapping has now been moved up a layer, it would > >> be easier to manage this mapping from higher levels. For example, we > >> could allow drivers to specify a CPU affinity map for their interrupts, > >> or we could allow a userland tool to bind IRQs to specific CPUs. > > > > > > FYI, I think I would prefer the latter, as a sys admin knows that he's > > routing packets between two different network interfaces and thus wants > > those devices on separate CPUs (for example), whereas the driver writer > > isn't really in a position to know that sort of thing. > > > > Where this is really useful is for people developing FreeBSD-based > appliances that have very specific and fixed needs. Also, it's not so > much important which CPU gets the interrupt as it is which CPU runs the > ithread for that interrupt. I guess that you can get a little better > latency by preempting directly from the low-level interrupt handler into > the ithread, but I don't know if that is noticable noise above the cost > of the context switch and inevitable lock operations and contention > involved. > > Anyways, thanks for this work, it looks very promising. I figured that if we do provide a userland tool, that the interface it used would both bind the IRQ (to bind the low-level handler) as well as the ithread (using sched_bind()) to the given CPU. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org