From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 13:03:58 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 217C6106566B for ; Mon, 23 Apr 2012 13:03:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 7CCDE8FC08 for ; Mon, 23 Apr 2012 13:03:57 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3ND3iEV060688; Mon, 23 Apr 2012 16:03:44 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q3ND3hRj006679; Mon, 23 Apr 2012 16:03:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3ND3h1T006678; Mon, 23 Apr 2012 16:03:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 23 Apr 2012 16:03:43 +0300 From: Konstantin Belousov To: Fengwei yin Message-ID: <20120423130343.GT2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> <20120423113838.GT32749@zxy.spb.ru> <20120423120720.GS2358@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vtxr9MvRW4bhShyg" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-threads@freebsd.org, jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 13:03:58 -0000 --vtxr9MvRW4bhShyg Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote: > On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov > wrote: > > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote: > >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov w= rote: > >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: > >> > > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wrote: > >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > >> >> > > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > >> >> >> > > >> >> >> >> Hi list, > >> >> >> >> If this is not correct question on the list, please let me kn= ow and > >> >> >> >> sorry for noise. > >> >> >> >> > >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn= 't see > >> >> >> >> memory barrier used in libc. > >> >> >> >> How can we make sure it's safe on SMP arch? > >> >> >> > > >> >> >> > /usr/include/machine/atomic.h: > >> >> >> > > >> >> >> > #define mb() =9A =9A__asm __volatile("lock; addl $0,(%%esp)" := : : "memory") > >> >> >> > #define wmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : := : "memory") > >> >> >> > #define rmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : := : "memory") > >> >> >> > > >> >> >> > >> >> >> Thanks for the information. But it looks no body use it in libc. > >> >> > > >> >> > I think no body in libc need memory barrier: libc don't work with > >> >> > peripheral, for atomic opertions used different macros. > >> >> > >> >> If we check the usage of __sinit(), it is a typical singleton patte= rn which > >> >> needs memory barrier to make sure no potential SMP issue. > >> >> > >> >> Or did I miss something here? > >> > > >> > What architecture with cache incoherency and FreeBSD support? > >> > >> I suppose it's not related with cache inchoherency (I could be wrong). > >> It's related > >> with reorder of instruction by CPU. > >> > >> Here is the link talking about why need memory barrier for singleton: > >> http://www.oaklib.org/docs/oak/singleton.html > >> > >> x86 has strict memory model and may not suffer this kind of issue. But > >> ARM need to > >> take care of it IMHO. > > > > Please note that __sinit is idempotent, so double-initialization is not > > an issue there. The only possible problematic case would be other thread > > executing exit and not noticing non-NULL value for __cleanup while curr= ent > > thread just set it. > > > > I am not sure how much real this race is. Each call to _sinit() is imme= diately > > followed by a lock acquire, typically FLOCKFILE(), which enforces full = barrier > > semantic due to pthread_mutex_lock call. The exit() performs __cxa_fina= lize() > > call before checking __cleanup value, and __cxa_finalize() itself locks > > atexit_mutex. So the race is tiny and probably possible only for somewh= at > > buggy applications which perform exit() while there are stdio operations > > in progress. > > > > Also note that some functions assign to __cleanup unconditionally. > > > > Do you see any real issue due to non-synchronized access to __cleanup ? >=20 > No. I didn't see real issue. I am just reviewing the code. >=20 > If you don't think __sinit has issue, let's check another code: > line 68 in libc/stdio/fclose.c > line 133 in libc/stdio/findfp.c (function __sfp()) >=20 > Which is trying to free a fp slot by assign 0 to fp->_flags. But if > the instrucation > could be re-ordered, another CPU could see fp->_flags is assigned to 0 > before the > cleanup from line 57 to 67. >=20 > Let's say, if another CPU is in line 133 of __sfp(), it could see > fp->_flags become > 0 before it's aware of the cleanup (Line 57 to line 67 in > libc/stdio/fclose.c) happen. >=20 > Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c > just could make sure > line 70 happen after line 68. It can't impact the re-order of line 57 > ~ line 68 by CPU. Yes, FUNLOCKFILE() there would have no effect on the potential CPU reorderi= ng of the writes. But does the order of these writes matter at all ? Please note that __sfp() reinitializes all fields written by fclose(). Only if CPU executing fclose() is allowed to reorder operations so that the external effect of _flags =3D 0 assignment can be observed before that CPU executes other operations from fclose(), there could be a problem. This is definitely impossible on Intel, and I indeed do not know about other architectures enough to reject such possibility. The _flags member is short, so atomics cannot be used there. The easier solution, if this is indeed an issue, is to lock thread_lock around _flags =3D 0 assignment in fclose(). --vtxr9MvRW4bhShyg Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+VUy8ACgkQC3+MBN1Mb4g7pwCgnxQskpojwuEEp4/fxAeP287y 6CEAniYTpQJ3iblCQ8JnmF5i0HJOx7te =ijpf -----END PGP SIGNATURE----- --vtxr9MvRW4bhShyg--