From owner-freebsd-arch@FreeBSD.ORG Mon Jan 21 09:54:59 2013 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AA844622; Mon, 21 Jan 2013 09:54:59 +0000 (UTC) (envelope-from marius@alchemy.franken.de) Received: from alchemy.franken.de (alchemy.franken.de [194.94.249.214]) by mx1.freebsd.org (Postfix) with ESMTP id 0743CB81; Mon, 21 Jan 2013 09:54:58 +0000 (UTC) Received: from alchemy.franken.de (localhost [127.0.0.1]) by alchemy.franken.de (8.14.5/8.14.5/ALCHEMY.FRANKEN.DE) with ESMTP id r0L9svRW099017; Mon, 21 Jan 2013 10:54:57 +0100 (CET) (envelope-from marius@alchemy.franken.de) Received: (from marius@localhost) by alchemy.franken.de (8.14.5/8.14.5/Submit) id r0L9sv57099016; Mon, 21 Jan 2013 10:54:57 +0100 (CET) (envelope-from marius) Date: Mon, 21 Jan 2013 10:54:57 +0100 From: Marius Strobl To: Alexander Motin Subject: Re: [RFC/RFT] calloutng Message-ID: <20130121095457.GL85306@alchemy.franken.de> References: <50CCAB99.4040308@FreeBSD.org> <50CE5B54.3050905@FreeBSD.org> <50D03173.9080904@FreeBSD.org> <20121225232126.GA47692@alchemy.franken.de> <50DB4EFE.2020600@FreeBSD.org> <20130106152313.GD26039@alchemy.franken.de> <50EBF921.2000304@FreeBSD.org> <20130113180940.GM26039@alchemy.franken.de> <50F30CAB.3000001@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <50F30CAB.3000001@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Davide Italiano , FreeBSD Current , freebsd-arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jan 2013 09:54:59 -0000 On Sun, Jan 13, 2013 at 09:36:11PM +0200, Alexander Motin wrote: > On 13.01.2013 20:09, Marius Strobl wrote: > > On Tue, Jan 08, 2013 at 12:46:57PM +0200, Alexander Motin wrote: > >> On 06.01.2013 17:23, Marius Strobl wrote: > >>> I'm not really sure what to do about that. Earlier you already said > >>> that sched_bind(9) also isn't an option in case if td_critnest > 1. > >>> To be honest, I don't really unerstand why using a spin lock in the > >>> timecounter path makes sparc64 the only problematic architecture > >>> for your changes. The x86 i8254_get_timecount() also uses a spin lock > >>> so it should be in the same boat. > >> > >> The problem is not in using spinlock, but in waiting for other CPU while > >> spinlock is held. Other CPU may also hold spinlock and wait for > >> something, causing deadlock. i8254 code uses spinlock just to atomically > >> access hardware registers, so it causes no problems. > > > > Okay, but wouldn't that be a general problem then? Pretty much > > anything triggering an IPI holds smp_ipi_mtx while doing so and > > the lower level IPI stuff waits for other CPU(s), including on > > x86. > > The problem is general. But now it works because single smp_ipi_mtx is > used in all cases where IPI result is waited. As soon as spinning > happens with interrupts still enabled, there is no deadlocks. But > problem reappears if any different lock is used, or locks are nested. I'm having a hard time getting an alternate time counter device to work. The crystal required for the counters in the south bridge just doesn't seem to be mounted any where near it (I've not looked at the bottom of the PCB though). While the time counter part of the on- board bge(4) driven chips basically work, they don't seem to like concurrent accesses caused by the rest of bge(4). I.e. although the counter is just read, sooner or later this causes a fatal bus error. I haven't tried serializing accesses to the chip, but getting to such a complexity for just reading a non-indexed register at least doesn't feel good ... However, AFAICT the scenario you describe can't happen. On sparc64, spinlock_enter() only raises the processor interrupt level, which doesn't block the direct cross traps I've implemented remote reading of (S)TICK as (which also means that the actions such traps may perform are very limitted and must occur in interrupt context, but which are sufficient for this purpose and in turn makes them very fast). I.e. although the AP holds smp_ipi_mtx or any amount of nested spin locks, this will not deadlock in case the BSP also holds any spin lock when reading (S)TICK from it. Marius