From owner-freebsd-arch Sun Mar 18 22:13:37 2001 Delivered-To: freebsd-arch@freebsd.org Received: from earth.backplane.com (earth-nat-cw.backplane.com [208.161.114.67]) by hub.freebsd.org (Postfix) with ESMTP id A9C2137B718; Sun, 18 Mar 2001 22:13:34 -0800 (PST) (envelope-from dillon@earth.backplane.com) Received: (from dillon@localhost) by earth.backplane.com (8.11.2/8.9.3) id f2J6DWH04251; Sun, 18 Mar 2001 22:13:32 -0800 (PST) (envelope-from dillon) Date: Sun, 18 Mar 2001 22:13:32 -0800 (PST) From: Matt Dillon Message-Id: <200103190613.f2J6DWH04251@earth.backplane.com> To: Bruce Evans Cc: John Baldwin , Matthew Jacob , arch@FreeBSD.ORG Subject: Re: 'final' man pages References: Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG I should make an addendum here, since BDE indicated 'by masks'. What I forgot to say was that, of course, for a mask-based spl mechanism to work without a giant lock, the mask needs to be per-cpu and the interrupt code either needs to check all the cpu's masks, or the aggregate needs to be cached. Caching the aggregate is actually fairly easy to do. You can simply OR-in to a single common global when adding mask elements. But you take a hit when removing mask elements (splx() equivalent). You have to iterate through all the cpu's or have some sort of master lock, which can be expensive. There is a way to do it using only bus locking, and that is to aggregate the multiple cpu's masks for any given interrupt number together into a single word. For example, if you had 4 cpu's each interrupt would eat 4 bits. Each cpu would have to use several locked OR instructions to set their individual bits and AND to clear them (because the 'mask' now spans multiple words), but the interrupt can use a single memory access to test all the per-cpu masks for a particular irq all at once, since they all reside in the same word. Another requirement when using a software interrupt mask is that you need an interrupt-in-progress mask as well as an interrupt-disable-mask. The spl*() function equivalent would have to set the interrupt-disable-mask and then spin until any running interrupt (on another cpu) completes by waiting for the equivalent interrupt-in-progress mask to clear. Once clear, the disable mask prevents further interrupts from occuring. This was the direction I proposed for SMPng as an intermediate step before going to mutexes for the interrupt subsystem but SMPng decided to go directly to the MUTEXes. Theoretically the MUTEX system is supposed to do away with this sort of masking requirement, but it requires serious work on the interrupt subsystem and mainline kernel code to operate as expected since most mainline code still assumes a mask/spl*() API. This is why current is still disabling interrupts in SMP unsafe code. If people recall, this led to a number of minor bugs rearing their heads like the FSYNC code getting into a cpu-bound loop waiting for async I/O ops to finish, etc. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message