From owner-freebsd-current@FreeBSD.ORG Tue May 17 13:46:38 2011 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4BA05106568A; Tue, 17 May 2011 13:46:38 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id F002D8FC12; Tue, 17 May 2011 13:46:36 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id QAA27157; Tue, 17 May 2011 16:46:35 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4DD27C3A.3040509@FreeBSD.org> Date: Tue, 17 May 2011 16:46:34 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110504 Lightning/1.0b2 Thunderbird/3.1.10 MIME-Version: 1.0 To: John Baldwin References: <4DCD357D.6000109@FreeBSD.org> <201105161421.27665.jhb@freebsd.org> <4DD17AB3.1070606@FreeBSD.org> <201105161609.21898.jhb@freebsd.org> <4DD22BD9.6070504@FreeBSD.org> <4DD26256.2070008@FreeBSD.org> In-Reply-To: <4DD26256.2070008@FreeBSD.org> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Max Laier , FreeBSD current , neel@FreeBSD.org, Peter Grehan Subject: Re: proposed smp_rendezvous change X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 May 2011 13:46:38 -0000 on 17/05/2011 14:56 John Baldwin said the following: > On 5/17/11 4:03 AM, Andriy Gapon wrote: >> Couldn't [Shouldn't] the whole: >> >>>>> /* Ensure we have up-to-date values. */ >>>>> atomic_add_acq_int(&smp_rv_waiters[0], 1); >>>>> while (smp_rv_waiters[0]< smp_rv_ncpus) >>>>> cpu_spinwait(); >> >> be just replaced with: >> >> rmb(); >> >> Or a proper MI function that does just a read memory barrier, if rmb() is not that. > > No, you could replace it with: > > atomic_add_acq_int(&smp_rv_waiters[0], 1); What about (void)atomic_load_acq(&smp_rv_waiters[0]); In my opinion that should ensure that the hardware must post the latest value from a master CPU to memory of smp_rv_waiters[0] and a slave CPU gets it from there. And also, because of memory barriers inserted by store_rel on the master CPU and load_acq on the slave CPU, the latest values of all other smp_rv_* fields should become visible to the slave CPU. > The key being that atomic_add_acq_int() will block (either in hardware or > software) until it can safely perform the atomic operation. That means waiting > until the write to set smp_rv_waiters[0] to 0 by the rendezvous initiator is > visible to the current CPU. > > On some platforms a write by one CPU may not post instantly to other CPUs (e.g. it > may sit in a store buffer). That is fine so long as an attempt to update that > value atomically (using cas or a conditional-store, etc.) fails. For those > platforms, the atomic(9) API is required to spin until it succeeds. > > This is why the mtx code spins if it can't set MTX_CONTESTED for example. > Thank you for the great explanation! Taking sparc64 as an example, I think that atomic_load_acq uses a degenerate cas call, which should take care of hardware synchronization. -- Andriy Gapon