From owner-freebsd-threads@FreeBSD.ORG Sun Sep 21 21:37:45 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AFD73914; Sun, 21 Sep 2014 21:37:45 +0000 (UTC) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 755B2C3E; Sun, 21 Sep 2014 21:37:45 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 7A079358C54; Sun, 21 Sep 2014 23:37:42 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 4287F28494; Sun, 21 Sep 2014 23:37:42 +0200 (CEST) Date: Sun, 21 Sep 2014 23:37:42 +0200 From: Jilles Tjoelker To: freebsd-threads@freebsd.org Subject: sem_post() performance Message-ID: <20140921213742.GA46868@stack.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Cc: adrian@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 21:37:45 -0000 It has been reported that POSIX semaphores are slow, in contexts such as Python. Note that POSIX semaphores are the only synchronization objects that support use by different processes in shared memory; this does not work for mutexes and condition variables because they are pointers to the actual data structure. In fact, sem_post() unconditionally performs an umtx system call. To avoid both lost wakeups and possible writes to a destroyed semaphore, an uncontested sem_post() must check the _has_waiters flag atomically with incrementing _count. The proper way to do this would be to take one bit from _count and use it for the _has_waiters flag; the definition of SEM_VALUE_MAX permits this. However, this would require a new set of umtx semaphore operations and will break ABI of process-shared semaphores (things may break if an old and a new libc access the same semaphore over shared memory). This diff only affects 32-bit aligned but 64-bit misaligned semaphores on 64-bit systems, and changes _count and _has_waiters atomically using a 64-bit atomic operation. It probably needs a may_alias attribute for correctness, but does not have a wrapper for that. Some x86 CPUs may cope with misaligned atomic ops without destroying performance (especially if they do not cross a cache line), so the alignment restriction could be relaxed to make the patch more practical. Many CPUs in the i386 architecture have a 64-bit atomic op (cmpxchg8b) which could be used here. This appears to restore performance of 10-stable uncontested semaphores with the strange alignment to 9-stable levels (a tight loop with sem_wait and sem_post). I have not tested in any real workload. Index: lib/libc/gen/sem_new.c =================================================================== --- lib/libc/gen/sem_new.c (revision 269952) +++ lib/libc/gen/sem_new.c (working copy) @@ -437,6 +437,32 @@ _sem_post(sem_t *sem) if (sem_check_validity(sem) != 0) return (-1); +#ifdef __LP64__ + if (((uintptr_t)&sem->_kern._count & 7) == 0) { + uint64_t oldval, newval; + + while (!sem->_kern._has_waiters) { + count = sem->_kern._count; + if (count + 1 > SEM_VALUE_MAX) + return (EOVERFLOW); + /* + * Expect _count == count and _has_waiters == 0. + */ +#if BYTE_ORDER == LITTLE_ENDIAN + oldval = (uint64_t)count << 32; + newval = (uint64_t)(count + 1) << 32; +#elif BYTE_ORDER == BIG_ENDIAN + oldval = (uint64_t)count; + newval = (uint64_t)(count + 1); +#else +#error Unknown byte order +#endif + if (atomic_cmpset_rel_64((uint64_t *)&sem->_kern._count, + oldval, newval)) + return (0); + } + } +#endif do { count = sem->_kern._count; if (count + 1 > SEM_VALUE_MAX) -- Jilles Tjoelker From owner-freebsd-threads@FreeBSD.ORG Mon Sep 22 20:36:13 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 55B8BF70; Mon, 22 Sep 2014 20:36:13 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2AA27386; Mon, 22 Sep 2014 20:36:13 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3A168B953; Mon, 22 Sep 2014 16:36:12 -0400 (EDT) From: John Baldwin To: freebsd-threads@freebsd.org Subject: Re: sem_post() performance Date: Mon, 22 Sep 2014 15:53:13 -0400 Message-ID: <1531724.MPBlj40xOW@ralph.baldwin.cx> User-Agent: KMail/4.10.5 (FreeBSD/10.0-STABLE; KDE/4.10.5; amd64; ; ) In-Reply-To: <20140921213742.GA46868@stack.nl> References: <20140921213742.GA46868@stack.nl> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 22 Sep 2014 16:36:12 -0400 (EDT) Cc: adrian@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 20:36:13 -0000 On Sunday, September 21, 2014 11:37:42 PM Jilles Tjoelker wrote: > It has been reported that POSIX semaphores are slow, in contexts such as > Python. Note that POSIX semaphores are the only synchronization objects > that support use by different processes in shared memory; this does not > work for mutexes and condition variables because they are pointers to > the actual data structure. > > In fact, sem_post() unconditionally performs an umtx system call. *sigh* I was worried that that might be the case. > To avoid both lost wakeups and possible writes to a destroyed semaphore, > an uncontested sem_post() must check the _has_waiters flag atomically > with incrementing _count. > > The proper way to do this would be to take one bit from _count and use > it for the _has_waiters flag; the definition of SEM_VALUE_MAX permits > this. However, this would require a new set of umtx semaphore operations > and will break ABI of process-shared semaphores (things may break if an > old and a new libc access the same semaphore over shared memory). > > This diff only affects 32-bit aligned but 64-bit misaligned semaphores > on 64-bit systems, and changes _count and _has_waiters atomically using > a 64-bit atomic operation. It probably needs a may_alias attribute for > correctness, but does not have a wrapper for that. It wasn't clear on first reading, but you are using aliasing to get around the need for new umtx calls by using a 64-bit atomic op to adjust two ints at the same time, yes? Note that since a failing semaphore op calls into the kernel for the "hard" case, you might in fact be able to change the ABI without breaking process-shared semaphores. That is, suppose you left 'has_waiters' as always true and reused the high bit of count for has_waiters. Would old binaries always trap into the kernel? (Not sure they will, especially the case where an old binary creates the semaphore, a new binary would have to force has_waiters to true in every sem op, but even that might not be enough.) > Some x86 CPUs may cope with misaligned atomic ops without destroying > performance (especially if they do not cross a cache line), so the > alignment restriction could be relaxed to make the patch more practical. > > Many CPUs in the i386 architecture have a 64-bit atomic op (cmpxchg8b) > which could be used here. > > This appears to restore performance of 10-stable uncontested semaphores > with the strange alignment to 9-stable levels (a tight loop with > sem_wait and sem_post). I have not tested in any real workload. > > Index: lib/libc/gen/sem_new.c > =================================================================== > --- lib/libc/gen/sem_new.c (revision 269952) > +++ lib/libc/gen/sem_new.c (working copy) > @@ -437,6 +437,32 @@ _sem_post(sem_t *sem) > if (sem_check_validity(sem) != 0) > return (-1); > > +#ifdef __LP64__ > + if (((uintptr_t)&sem->_kern._count & 7) == 0) { > + uint64_t oldval, newval; > + > + while (!sem->_kern._has_waiters) { > + count = sem->_kern._count; > + if (count + 1 > SEM_VALUE_MAX) > + return (EOVERFLOW); > + /* > + * Expect _count == count and _has_waiters == 0. > + */ > +#if BYTE_ORDER == LITTLE_ENDIAN > + oldval = (uint64_t)count << 32; > + newval = (uint64_t)(count + 1) << 32; > +#elif BYTE_ORDER == BIG_ENDIAN > + oldval = (uint64_t)count; > + newval = (uint64_t)(count + 1); > +#else > +#error Unknown byte order > +#endif > + if (atomic_cmpset_rel_64((uint64_t *)&sem->_kern._count, > + oldval, newval)) > + return (0); > + } > + } > +#endif I think this looks ok, but it's a shame we can't fix it more cleanly. -- John Baldwin From owner-freebsd-threads@FreeBSD.ORG Tue Sep 23 21:20:06 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8EB51B3; Tue, 23 Sep 2014 21:20:06 +0000 (UTC) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3C0D2AB6; Tue, 23 Sep 2014 21:20:06 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id A807C3592E6; Tue, 23 Sep 2014 23:20:00 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 8C51128494; Tue, 23 Sep 2014 23:20:00 +0200 (CEST) Date: Tue, 23 Sep 2014 23:20:00 +0200 From: Jilles Tjoelker To: John Baldwin Subject: Re: sem_post() performance Message-ID: <20140923212000.GA78110@stack.nl> References: <20140921213742.GA46868@stack.nl> <1531724.MPBlj40xOW@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1531724.MPBlj40xOW@ralph.baldwin.cx> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: adrian@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 21:20:06 -0000 On Mon, Sep 22, 2014 at 03:53:13PM -0400, John Baldwin wrote: > On Sunday, September 21, 2014 11:37:42 PM Jilles Tjoelker wrote: > > It has been reported that POSIX semaphores are slow, in contexts such as > > Python. Note that POSIX semaphores are the only synchronization objects > > that support use by different processes in shared memory; this does not > > work for mutexes and condition variables because they are pointers to > > the actual data structure. > > In fact, sem_post() unconditionally performs an umtx system call. > *sigh* I was worried that that might be the case. > > To avoid both lost wakeups and possible writes to a destroyed semaphore, > > an uncontested sem_post() must check the _has_waiters flag atomically > > with incrementing _count. > > The proper way to do this would be to take one bit from _count and > > use it for the _has_waiters flag; the definition of SEM_VALUE_MAX > > permits this. However, this would require a new set of umtx > > semaphore operations and will break ABI of process-shared semaphores > > (things may break if an old and a new libc access the same semaphore > > over shared memory). > > This diff only affects 32-bit aligned but 64-bit misaligned > > semaphores on 64-bit systems, and changes _count and _has_waiters > > atomically using a 64-bit atomic operation. It probably needs a > > may_alias attribute for correctness, but does not have > > a wrapper for that. > It wasn't clear on first reading, but you are using aliasing to get > around the need for new umtx calls by using a 64-bit atomic op to > adjust two ints at the same time, yes? Note that since a failing > semaphore op calls into the kernel for the "hard" case, you might in > fact be able to change the ABI without breaking process-shared > semaphores. That is, suppose you left 'has_waiters' as always true > and reused the high bit of count for has_waiters. > Would old binaries always trap into the kernel? (Not sure they will, > especially the case where an old binary creates the semaphore, a new > binary would have to force has_waiters to true in every sem op, but > even that might not be enough.) I think that everything will break when a binary linked to old and new libcs use the same semaphore. If the new contested bit is set, the old sem_getvalue() will return garbage, the old sem_trywait() will fail even if the real count is greater than 0, the old sem_wait() and sem_timedwait() may spin if the real count is greater than 0 and the old sem_post() will fail with [EOVERFLOW]. That the "hard" path always issues a system call does not help much, since the system calls do not write to _count (this is an throughput optimization, allowing a fast-path thread through while a slow-path thread is entering or leaving the kernel). > > Some x86 CPUs may cope with misaligned atomic ops without destroying > > performance (especially if they do not cross a cache line), so the > > alignment restriction could be relaxed to make the patch more practical. > > Many CPUs in the i386 architecture have a 64-bit atomic op (cmpxchg8b) > > which could be used here. > > This appears to restore performance of 10-stable uncontested semaphores > > with the strange alignment to 9-stable levels (a tight loop with > > sem_wait and sem_post). I have not tested in any real workload. > > Index: lib/libc/gen/sem_new.c > > =================================================================== > > --- lib/libc/gen/sem_new.c (revision 269952) > > +++ lib/libc/gen/sem_new.c (working copy) > > @@ -437,6 +437,32 @@ _sem_post(sem_t *sem) > > if (sem_check_validity(sem) != 0) > > return (-1); > > > > +#ifdef __LP64__ > > + if (((uintptr_t)&sem->_kern._count & 7) == 0) { > > + uint64_t oldval, newval; > > + > > + while (!sem->_kern._has_waiters) { > > + count = sem->_kern._count; > > + if (count + 1 > SEM_VALUE_MAX) > > + return (EOVERFLOW); > > + /* > > + * Expect _count == count and _has_waiters == 0. > > + */ > > +#if BYTE_ORDER == LITTLE_ENDIAN > > + oldval = (uint64_t)count << 32; > > + newval = (uint64_t)(count + 1) << 32; > > +#elif BYTE_ORDER == BIG_ENDIAN > > + oldval = (uint64_t)count; > > + newval = (uint64_t)(count + 1); > > +#else > > +#error Unknown byte order > > +#endif > > + if (atomic_cmpset_rel_64((uint64_t *)&sem->_kern._count, > > + oldval, newval)) > > + return (0); > > + } > > + } > > +#endif > I think this looks ok, but it's a shame we can't fix it more cleanly. I don't like at all that you have to either force an exotic alignment or use unaligned atomic ops. I saw another interesting sem_ implementation in musl libc. It also uses a separate waiters word (but an int instead of a boolean), and is based on the futex calls (which we already have as wait_uint/wake umtx calls). It looks rather complicated but the musl libc people are usually good at getting this kind of stuff right. I suppose this will also break subtly when interoperating with old libc, and adding new umtx ops will simplify the algorithm. Consideration: just declare mixing process-shared semaphores with sufficiently different libc unsupported, and change SEM_MAGIC to enforce that? (This does not prevent running old binaries, as long as they're dynamically linked to libc and you use a new libc.so.) Consideration 2: use the new implementation only for process-private semaphores and keep using the old slow one for process-shared semaphores? -- Jilles Tjoelker From owner-freebsd-threads@FreeBSD.ORG Tue Sep 23 21:42:29 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8707E6DE; Tue, 23 Sep 2014 21:42:29 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 72F94D4B; Tue, 23 Sep 2014 21:42:29 +0000 (UTC) Received: from u10-2-32-011.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id 73F2E346DE75; Tue, 23 Sep 2014 14:42:28 -0700 (PDT) Message-ID: <5421E943.3010107@mu.org> Date: Tue, 23 Sep 2014 14:42:27 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: Jilles Tjoelker , John Baldwin Subject: Re: sem_post() performance References: <20140921213742.GA46868@stack.nl> <1531724.MPBlj40xOW@ralph.baldwin.cx> <20140923212000.GA78110@stack.nl> In-Reply-To: <20140923212000.GA78110@stack.nl> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: adrian@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 21:42:29 -0000 On 9/23/14 2:20 PM, Jilles Tjoelker wrote: > On Mon, Sep 22, 2014 at 03:53:13PM -0400, John Baldwin wrote: >> On Sunday, September 21, 2014 11:37:42 PM Jilles Tjoelker wrote: >>> It has been reported that POSIX semaphores are slow, in contexts such as >>> Python. Note that POSIX semaphores are the only synchronization objects >>> that support use by different processes in shared memory; this does not >>> work for mutexes and condition variables because they are pointers to >>> the actual data structure. >>> In fact, sem_post() unconditionally performs an umtx system call. >> *sigh* I was worried that that might be the case. >>> To avoid both lost wakeups and possible writes to a destroyed semaphore, >>> an uncontested sem_post() must check the _has_waiters flag atomically >>> with incrementing _count. >>> The proper way to do this would be to take one bit from _count and >>> use it for the _has_waiters flag; the definition of SEM_VALUE_MAX >>> permits this. However, this would require a new set of umtx >>> semaphore operations and will break ABI of process-shared semaphores >>> (things may break if an old and a new libc access the same semaphore >>> over shared memory). >>> This diff only affects 32-bit aligned but 64-bit misaligned >>> semaphores on 64-bit systems, and changes _count and _has_waiters >>> atomically using a 64-bit atomic operation. It probably needs a >>> may_alias attribute for correctness, but does not have >>> a wrapper for that. >> It wasn't clear on first reading, but you are using aliasing to get >> around the need for new umtx calls by using a 64-bit atomic op to >> adjust two ints at the same time, yes? Note that since a failing >> semaphore op calls into the kernel for the "hard" case, you might in >> fact be able to change the ABI without breaking process-shared >> semaphores. That is, suppose you left 'has_waiters' as always true >> and reused the high bit of count for has_waiters. >> Would old binaries always trap into the kernel? (Not sure they will, >> especially the case where an old binary creates the semaphore, a new >> binary would have to force has_waiters to true in every sem op, but >> even that might not be enough.) > I think that everything will break when a binary linked to old and new > libcs use the same semaphore. If the new contested bit is set, the old > sem_getvalue() will return garbage, the old sem_trywait() will fail even > if the real count is greater than 0, the old sem_wait() and > sem_timedwait() may spin if the real count is greater than 0 and the old > sem_post() will fail with [EOVERFLOW]. > > That the "hard" path always issues a system call does not help much, > since the system calls do not write to _count (this is an throughput > optimization, allowing a fast-path thread through while a slow-path > thread is entering or leaving the kernel). > >>> Some x86 CPUs may cope with misaligned atomic ops without destroying >>> performance (especially if they do not cross a cache line), so the >>> alignment restriction could be relaxed to make the patch more practical. >>> Many CPUs in the i386 architecture have a 64-bit atomic op (cmpxchg8b) >>> which could be used here. >>> This appears to restore performance of 10-stable uncontested semaphores >>> with the strange alignment to 9-stable levels (a tight loop with >>> sem_wait and sem_post). I have not tested in any real workload. >>> Index: lib/libc/gen/sem_new.c >>> =================================================================== >>> --- lib/libc/gen/sem_new.c (revision 269952) >>> +++ lib/libc/gen/sem_new.c (working copy) >>> @@ -437,6 +437,32 @@ _sem_post(sem_t *sem) >>> if (sem_check_validity(sem) != 0) >>> return (-1); >>> >>> +#ifdef __LP64__ >>> + if (((uintptr_t)&sem->_kern._count & 7) == 0) { >>> + uint64_t oldval, newval; >>> + >>> + while (!sem->_kern._has_waiters) { >>> + count = sem->_kern._count; >>> + if (count + 1 > SEM_VALUE_MAX) >>> + return (EOVERFLOW); >>> + /* >>> + * Expect _count == count and _has_waiters == 0. >>> + */ >>> +#if BYTE_ORDER == LITTLE_ENDIAN >>> + oldval = (uint64_t)count << 32; >>> + newval = (uint64_t)(count + 1) << 32; >>> +#elif BYTE_ORDER == BIG_ENDIAN >>> + oldval = (uint64_t)count; >>> + newval = (uint64_t)(count + 1); >>> +#else >>> +#error Unknown byte order >>> +#endif >>> + if (atomic_cmpset_rel_64((uint64_t *)&sem->_kern._count, >>> + oldval, newval)) >>> + return (0); >>> + } >>> + } >>> +#endif >> I think this looks ok, but it's a shame we can't fix it more cleanly. > I don't like at all that you have to either force an exotic alignment or > use unaligned atomic ops. > > I saw another interesting sem_ implementation in musl libc. It also uses > a separate waiters word (but an int instead of a boolean), and is based > on the futex calls (which we already have as wait_uint/wake umtx calls). > It looks rather complicated but the musl libc people are usually good at > getting this kind of stuff right. I suppose this will also break subtly > when interoperating with old libc, and adding new umtx ops will simplify > the algorithm. > > Consideration: just declare mixing process-shared semaphores with > sufficiently different libc unsupported, and change SEM_MAGIC to enforce > that? (This does not prevent running old binaries, as long as they're > dynamically linked to libc and you use a new libc.so.) > > > Consideration 2: use the new implementation only for process-private > semaphores and keep using the old slow one for process-shared > semaphores? > I believe a flag day for "new/old" mutexes should be OK. (Consideration #1) Would it be possible to compile world/kernel to keep old compat for those that *really* want to mix libraries and whatnot? imo, if people really wanted to, we could backpatch old libc and distrib that as its own package. Then we only break people derp enough to use .a files. -Alfred From owner-freebsd-threads@FreeBSD.ORG Tue Sep 23 21:49:36 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 01BC284B; Tue, 23 Sep 2014 21:49:36 +0000 (UTC) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.netplex.net", Issuer "RapidSSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B0851D91; Tue, 23 Sep 2014 21:49:35 +0000 (UTC) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.9/8.14.9/NETPLEX) with ESMTP id s8NLnSMZ013502; Tue, 23 Sep 2014 17:49:28 -0400 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.3 (mail.netplex.net [204.213.176.9]); Tue, 23 Sep 2014 17:49:28 -0400 (EDT) Date: Tue, 23 Sep 2014 17:49:28 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net Reply-To: Daniel Eischen To: Jilles Tjoelker Subject: Re: sem_post() performance In-Reply-To: <20140923212000.GA78110@stack.nl> Message-ID: References: <20140921213742.GA46868@stack.nl> <1531724.MPBlj40xOW@ralph.baldwin.cx> <20140923212000.GA78110@stack.nl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: adrian@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 21:49:36 -0000 On Tue, 23 Sep 2014, Jilles Tjoelker wrote: > On Mon, Sep 22, 2014 at 03:53:13PM -0400, John Baldwin wrote: >> On Sunday, September 21, 2014 11:37:42 PM Jilles Tjoelker wrote: >>> It has been reported that POSIX semaphores are slow, in contexts such as >>> Python. Note that POSIX semaphores are the only synchronization objects >>> that support use by different processes in shared memory; this does not >>> work for mutexes and condition variables because they are pointers to >>> the actual data structure. > >>> In fact, sem_post() unconditionally performs an umtx system call. > >> *sigh* I was worried that that might be the case. > >>> To avoid both lost wakeups and possible writes to a destroyed semaphore, >>> an uncontested sem_post() must check the _has_waiters flag atomically >>> with incrementing _count. > >>> The proper way to do this would be to take one bit from _count and >>> use it for the _has_waiters flag; the definition of SEM_VALUE_MAX >>> permits this. However, this would require a new set of umtx >>> semaphore operations and will break ABI of process-shared semaphores >>> (things may break if an old and a new libc access the same semaphore >>> over shared memory). > >>> This diff only affects 32-bit aligned but 64-bit misaligned >>> semaphores on 64-bit systems, and changes _count and _has_waiters >>> atomically using a 64-bit atomic operation. It probably needs a >>> may_alias attribute for correctness, but does not have >>> a wrapper for that. > >> It wasn't clear on first reading, but you are using aliasing to get >> around the need for new umtx calls by using a 64-bit atomic op to >> adjust two ints at the same time, yes? Note that since a failing >> semaphore op calls into the kernel for the "hard" case, you might in >> fact be able to change the ABI without breaking process-shared >> semaphores. That is, suppose you left 'has_waiters' as always true >> and reused the high bit of count for has_waiters. > >> Would old binaries always trap into the kernel? (Not sure they will, >> especially the case where an old binary creates the semaphore, a new >> binary would have to force has_waiters to true in every sem op, but >> even that might not be enough.) > > I think that everything will break when a binary linked to old and new > libcs use the same semaphore. If the new contested bit is set, the old > sem_getvalue() will return garbage, the old sem_trywait() will fail even > if the real count is greater than 0, the old sem_wait() and > sem_timedwait() may spin if the real count is greater than 0 and the old > sem_post() will fail with [EOVERFLOW]. > > That the "hard" path always issues a system call does not help much, > since the system calls do not write to _count (this is an throughput > optimization, allowing a fast-path thread through while a slow-path > thread is entering or leaving the kernel). [ ... ] > Consideration: just declare mixing process-shared semaphores with > sufficiently different libc unsupported, and change SEM_MAGIC to enforce > that? (This does not prevent running old binaries, as long as they're > dynamically linked to libc and you use a new libc.so.) Yes and yes :-) And we need to add such a magic or version number to our mutex and CVs when we convert their types from pointers to actual structs. -- DE From owner-freebsd-threads@FreeBSD.ORG Wed Sep 24 14:12:05 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2E78777D; Wed, 24 Sep 2014 14:12:05 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DD1DD195; Wed, 24 Sep 2014 14:12:04 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 56AC1B91F; Wed, 24 Sep 2014 10:12:03 -0400 (EDT) From: John Baldwin To: Jilles Tjoelker Subject: Re: sem_post() performance Date: Wed, 24 Sep 2014 10:11:51 -0400 Message-ID: <2626215.ysfmRFbjz5@ralph.baldwin.cx> User-Agent: KMail/4.10.5 (FreeBSD/10.0-STABLE; KDE/4.10.5; amd64; ; ) In-Reply-To: <20140923212000.GA78110@stack.nl> References: <20140921213742.GA46868@stack.nl> <1531724.MPBlj40xOW@ralph.baldwin.cx> <20140923212000.GA78110@stack.nl> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 24 Sep 2014 10:12:03 -0400 (EDT) Cc: adrian@freebsd.org, kib@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 14:12:05 -0000 On Tuesday, September 23, 2014 11:20:00 PM Jilles Tjoelker wrote: > On Mon, Sep 22, 2014 at 03:53:13PM -0400, John Baldwin wrote: > > On Sunday, September 21, 2014 11:37:42 PM Jilles Tjoelker wrote: > > > It has been reported that POSIX semaphores are slow, in contexts such as > > > Python. Note that POSIX semaphores are the only synchronization objects > > > that support use by different processes in shared memory; this does not > > > work for mutexes and condition variables because they are pointers to > > > the actual data structure. > > > > > > In fact, sem_post() unconditionally performs an umtx system call. > > > > *sigh* I was worried that that might be the case. > > > > > To avoid both lost wakeups and possible writes to a destroyed semaphore, > > > an uncontested sem_post() must check the _has_waiters flag atomically > > > with incrementing _count. > > > > > > The proper way to do this would be to take one bit from _count and > > > use it for the _has_waiters flag; the definition of SEM_VALUE_MAX > > > permits this. However, this would require a new set of umtx > > > semaphore operations and will break ABI of process-shared semaphores > > > (things may break if an old and a new libc access the same semaphore > > > over shared memory). > > > > > > This diff only affects 32-bit aligned but 64-bit misaligned > > > semaphores on 64-bit systems, and changes _count and _has_waiters > > > atomically using a 64-bit atomic operation. It probably needs a > > > may_alias attribute for correctness, but does not have > > > a wrapper for that. > > > > It wasn't clear on first reading, but you are using aliasing to get > > around the need for new umtx calls by using a 64-bit atomic op to > > adjust two ints at the same time, yes? Note that since a failing > > semaphore op calls into the kernel for the "hard" case, you might in > > fact be able to change the ABI without breaking process-shared > > semaphores. That is, suppose you left 'has_waiters' as always true > > and reused the high bit of count for has_waiters. > > > > Would old binaries always trap into the kernel? (Not sure they will, > > especially the case where an old binary creates the semaphore, a new > > binary would have to force has_waiters to true in every sem op, but > > even that might not be enough.) > > I think that everything will break when a binary linked to old and new > libcs use the same semaphore. If the new contested bit is set, the old > sem_getvalue() will return garbage, the old sem_trywait() will fail even > if the real count is greater than 0, the old sem_wait() and > sem_timedwait() may spin if the real count is greater than 0 and the old > sem_post() will fail with [EOVERFLOW]. Well, keep in mind you can't have a binary linked to both libc's generally (way too many things break if you do that, like passing the result of strdup() from one libc to free() of the other, etc.). The real problem case is multiple binaries. However, barring truly bizarre cases with people using funky LD_LIBRARY_PATH, at least one binary would have to be static to really get into trouble as in all but the bizarre cases all binaries will be sharing the same libc.so.7. However, it was more wishful thinking on my part that we could arrange things to force old binaries to still work (just slowly) while fixing things cleanly. > > I think this looks ok, but it's a shame we can't fix it more cleanly. > > I don't like at all that you have to either force an exotic alignment or > use unaligned atomic ops. > > I saw another interesting sem_ implementation in musl libc. It also uses > a separate waiters word (but an int instead of a boolean), and is based > on the futex calls (which we already have as wait_uint/wake umtx calls). > It looks rather complicated but the musl libc people are usually good at > getting this kind of stuff right. I suppose this will also break subtly > when interoperating with old libc, and adding new umtx ops will simplify > the algorithm. > > Consideration: just declare mixing process-shared semaphores with > sufficiently different libc unsupported, and change SEM_MAGIC to enforce > that? (This does not prevent running old binaries, as long as they're > dynamically linked to libc and you use a new libc.so.) This only breaks old static binaries (note that we only recently got pshared semaphores in 9.x or so). You could even allow partial compat for those if needed by recognizing the old SEM_MAGIC when opening an existing shared semaphore (so that compat would work so long as an old binary created the semaphore). That compat can even be added retroactively to libc if it turns out we need it. I probably think that is ok, but it would be an ABI breakage that would need to be documented. One thing that makes this slightly less painful is that new-style semaphores in 9.0+ are per-chroot, so if you have a jail that runs only old binaries, they will continue to work fine. I'm cc'ing Konstantin to see what he thinks. > Consideration 2: use the new implementation only for process-private > semaphores and keep using the old slow one for process-shared > semaphores? This is certainly safe, just requires extra work in the implementation. -- John Baldwin From owner-freebsd-threads@FreeBSD.ORG Wed Sep 24 14:31:11 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2204FE64; Wed, 24 Sep 2014 14:31:11 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C594334; Wed, 24 Sep 2014 14:31:10 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8OEV5oc022136 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Sep 2014 17:31:05 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8OEV5oc022136 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8OEV4GY022135; Wed, 24 Sep 2014 17:31:04 +0300 (EEST) (envelope-from kostik) Date: Wed, 24 Sep 2014 17:31:04 +0300 From: Konstantin Belousov To: John Baldwin Subject: Re: sem_post() performance Message-ID: <20140924143104.GK8870@kib.kiev.ua> References: <20140921213742.GA46868@stack.nl> <1531724.MPBlj40xOW@ralph.baldwin.cx> <20140923212000.GA78110@stack.nl> <2626215.ysfmRFbjz5@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2626215.ysfmRFbjz5@ralph.baldwin.cx> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: adrian@freebsd.org, kib@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 14:31:11 -0000 On Wed, Sep 24, 2014 at 10:11:51AM -0400, John Baldwin wrote: > On Tuesday, September 23, 2014 11:20:00 PM Jilles Tjoelker wrote: > > On Mon, Sep 22, 2014 at 03:53:13PM -0400, John Baldwin wrote: > > > On Sunday, September 21, 2014 11:37:42 PM Jilles Tjoelker wrote: > > > > It has been reported that POSIX semaphores are slow, in contexts such as > > > > Python. Note that POSIX semaphores are the only synchronization objects > > > > that support use by different processes in shared memory; this does not > > > > work for mutexes and condition variables because they are pointers to > > > > the actual data structure. > > > > > > > > In fact, sem_post() unconditionally performs an umtx system call. > > > > > > *sigh* I was worried that that might be the case. > > > > > > > To avoid both lost wakeups and possible writes to a destroyed semaphore, > > > > an uncontested sem_post() must check the _has_waiters flag atomically > > > > with incrementing _count. > > > > > > > > The proper way to do this would be to take one bit from _count and > > > > use it for the _has_waiters flag; the definition of SEM_VALUE_MAX > > > > permits this. However, this would require a new set of umtx > > > > semaphore operations and will break ABI of process-shared semaphores > > > > (things may break if an old and a new libc access the same semaphore > > > > over shared memory). > > > > > > > > This diff only affects 32-bit aligned but 64-bit misaligned > > > > semaphores on 64-bit systems, and changes _count and _has_waiters > > > > atomically using a 64-bit atomic operation. It probably needs a > > > > may_alias attribute for correctness, but does not have > > > > a wrapper for that. > > > > > > It wasn't clear on first reading, but you are using aliasing to get > > > around the need for new umtx calls by using a 64-bit atomic op to > > > adjust two ints at the same time, yes? Note that since a failing > > > semaphore op calls into the kernel for the "hard" case, you might in > > > fact be able to change the ABI without breaking process-shared > > > semaphores. That is, suppose you left 'has_waiters' as always true > > > and reused the high bit of count for has_waiters. > > > > > > Would old binaries always trap into the kernel? (Not sure they will, > > > especially the case where an old binary creates the semaphore, a new > > > binary would have to force has_waiters to true in every sem op, but > > > even that might not be enough.) > > > > I think that everything will break when a binary linked to old and new > > libcs use the same semaphore. If the new contested bit is set, the old > > sem_getvalue() will return garbage, the old sem_trywait() will fail even > > if the real count is greater than 0, the old sem_wait() and > > sem_timedwait() may spin if the real count is greater than 0 and the old > > sem_post() will fail with [EOVERFLOW]. > > Well, keep in mind you can't have a binary linked to both libc's generally > (way too many things break if you do that, like passing the result of strdup() > from one libc to free() of the other, etc.). The real problem case is > multiple binaries. However, barring truly bizarre cases with people using > funky LD_LIBRARY_PATH, at least one binary would have to be static to really > get into trouble as in all but the bizarre cases all binaries will be sharing > the same libc.so.7. > > However, it was more wishful thinking on my part that we could arrange things > to force old binaries to still work (just slowly) while fixing things cleanly. > > > > I think this looks ok, but it's a shame we can't fix it more cleanly. > > > > I don't like at all that you have to either force an exotic alignment or > > use unaligned atomic ops. > > > > I saw another interesting sem_ implementation in musl libc. It also uses > > a separate waiters word (but an int instead of a boolean), and is based > > on the futex calls (which we already have as wait_uint/wake umtx calls). > > It looks rather complicated but the musl libc people are usually good at > > getting this kind of stuff right. I suppose this will also break subtly > > when interoperating with old libc, and adding new umtx ops will simplify > > the algorithm. > > > > Consideration: just declare mixing process-shared semaphores with > > sufficiently different libc unsupported, and change SEM_MAGIC to enforce > > that? (This does not prevent running old binaries, as long as they're > > dynamically linked to libc and you use a new libc.so.) > > This only breaks old static binaries (note that we only recently got pshared > semaphores in 9.x or so). You could even allow partial compat for those if > needed by recognizing the old SEM_MAGIC when opening an existing shared > semaphore (so that compat would work so long as an old binary created the > semaphore). That compat can even be added retroactively to libc if it turns > out we need it. I probably think that is ok, but it would be an ABI breakage > that would need to be documented. One thing that makes this slightly less > painful is that new-style semaphores in 9.0+ are per-chroot, so if you have > a jail that runs only old binaries, they will continue to work fine. > > I'm cc'ing Konstantin to see what he thinks. You may also get in trouble if you upgrade libc without reboot, or use static binary built against newer libc on older host. But SEM_MAGIC provides at least protection against mixing incompatible implementations, and it seems that the checks are enough to ensure that both bad cases (new/old and old/new) are handled. Do we have a real use case when the old and new implementations have to be mixed ? Was it the same situation when the sem.c -> sem_new.c transition was made ? I think that the answer to the questions are no/yes, and we should just change magic and not worry about the issue until real use case appear. > > > Consideration 2: use the new implementation only for process-private > > semaphores and keep using the old slow one for process-shared > > semaphores? > > This is certainly safe, just requires extra work in the implementation. > > -- > John Baldwin From owner-freebsd-threads@FreeBSD.ORG Wed Sep 24 14:45:26 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0F3504E8; Wed, 24 Sep 2014 14:45:26 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 898977D2; Wed, 24 Sep 2014 14:45:25 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8OEjJOs022377 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Sep 2014 17:45:19 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8OEjJOs022377 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8OEjJvl022376; Wed, 24 Sep 2014 17:45:19 +0300 (EEST) (envelope-from kostik) Date: Wed, 24 Sep 2014 17:45:19 +0300 From: Konstantin Belousov To: John Baldwin Subject: Re: sem_post() performance Message-ID: <20140924144519.GL8870@kib.kiev.ua> References: <20140921213742.GA46868@stack.nl> <1531724.MPBlj40xOW@ralph.baldwin.cx> <20140923212000.GA78110@stack.nl> <2626215.ysfmRFbjz5@ralph.baldwin.cx> <20140924143104.GK8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924143104.GK8870@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: adrian@freebsd.org, kib@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 14:45:26 -0000 On Wed, Sep 24, 2014 at 05:31:04PM +0300, Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 10:11:51AM -0400, John Baldwin wrote: > > On Tuesday, September 23, 2014 11:20:00 PM Jilles Tjoelker wrote: > > > On Mon, Sep 22, 2014 at 03:53:13PM -0400, John Baldwin wrote: > > > > On Sunday, September 21, 2014 11:37:42 PM Jilles Tjoelker wrote: > > > > > It has been reported that POSIX semaphores are slow, in contexts such as > > > > > Python. Note that POSIX semaphores are the only synchronization objects > > > > > that support use by different processes in shared memory; this does not > > > > > work for mutexes and condition variables because they are pointers to > > > > > the actual data structure. > > > > > > > > > > In fact, sem_post() unconditionally performs an umtx system call. > > > > > > > > *sigh* I was worried that that might be the case. > > > > > > > > > To avoid both lost wakeups and possible writes to a destroyed semaphore, > > > > > an uncontested sem_post() must check the _has_waiters flag atomically > > > > > with incrementing _count. > > > > > > > > > > The proper way to do this would be to take one bit from _count and > > > > > use it for the _has_waiters flag; the definition of SEM_VALUE_MAX > > > > > permits this. However, this would require a new set of umtx > > > > > semaphore operations and will break ABI of process-shared semaphores > > > > > (things may break if an old and a new libc access the same semaphore > > > > > over shared memory). > > > > > > > > > > This diff only affects 32-bit aligned but 64-bit misaligned > > > > > semaphores on 64-bit systems, and changes _count and _has_waiters > > > > > atomically using a 64-bit atomic operation. It probably needs a > > > > > may_alias attribute for correctness, but does not have > > > > > a wrapper for that. > > > > > > > > It wasn't clear on first reading, but you are using aliasing to get > > > > around the need for new umtx calls by using a 64-bit atomic op to > > > > adjust two ints at the same time, yes? Note that since a failing > > > > semaphore op calls into the kernel for the "hard" case, you might in > > > > fact be able to change the ABI without breaking process-shared > > > > semaphores. That is, suppose you left 'has_waiters' as always true > > > > and reused the high bit of count for has_waiters. > > > > > > > > Would old binaries always trap into the kernel? (Not sure they will, > > > > especially the case where an old binary creates the semaphore, a new > > > > binary would have to force has_waiters to true in every sem op, but > > > > even that might not be enough.) > > > > > > I think that everything will break when a binary linked to old and new > > > libcs use the same semaphore. If the new contested bit is set, the old > > > sem_getvalue() will return garbage, the old sem_trywait() will fail even > > > if the real count is greater than 0, the old sem_wait() and > > > sem_timedwait() may spin if the real count is greater than 0 and the old > > > sem_post() will fail with [EOVERFLOW]. > > > > Well, keep in mind you can't have a binary linked to both libc's generally > > (way too many things break if you do that, like passing the result of strdup() > > from one libc to free() of the other, etc.). The real problem case is > > multiple binaries. However, barring truly bizarre cases with people using > > funky LD_LIBRARY_PATH, at least one binary would have to be static to really > > get into trouble as in all but the bizarre cases all binaries will be sharing > > the same libc.so.7. > > > > However, it was more wishful thinking on my part that we could arrange things > > to force old binaries to still work (just slowly) while fixing things cleanly. > > > > > > I think this looks ok, but it's a shame we can't fix it more cleanly. > > > > > > I don't like at all that you have to either force an exotic alignment or > > > use unaligned atomic ops. > > > > > > I saw another interesting sem_ implementation in musl libc. It also uses > > > a separate waiters word (but an int instead of a boolean), and is based > > > on the futex calls (which we already have as wait_uint/wake umtx calls). > > > It looks rather complicated but the musl libc people are usually good at > > > getting this kind of stuff right. I suppose this will also break subtly > > > when interoperating with old libc, and adding new umtx ops will simplify > > > the algorithm. > > > > > > Consideration: just declare mixing process-shared semaphores with > > > sufficiently different libc unsupported, and change SEM_MAGIC to enforce > > > that? (This does not prevent running old binaries, as long as they're > > > dynamically linked to libc and you use a new libc.so.) > > > > This only breaks old static binaries (note that we only recently got pshared > > semaphores in 9.x or so). You could even allow partial compat for those if > > needed by recognizing the old SEM_MAGIC when opening an existing shared > > semaphore (so that compat would work so long as an old binary created the > > semaphore). That compat can even be added retroactively to libc if it turns > > out we need it. I probably think that is ok, but it would be an ABI breakage > > that would need to be documented. One thing that makes this slightly less > > painful is that new-style semaphores in 9.0+ are per-chroot, so if you have > > a jail that runs only old binaries, they will continue to work fine. > > > > I'm cc'ing Konstantin to see what he thinks. > You may also get in trouble if you upgrade libc without reboot, or use > static binary built against newer libc on older host. > > But SEM_MAGIC provides at least protection against mixing incompatible > implementations, and it seems that the checks are enough to ensure > that both bad cases (new/old and old/new) are handled. > > Do we have a real use case when the old and new implementations have to > be mixed ? Was it the same situation when the sem.c -> sem_new.c transition > was made ? I think that the answer to the questions are no/yes, and we > should just change magic and not worry about the issue until real use > case appear. I think it is even worse now. If the application linked against FBSD_1.0 (ksem) semaphores implementation runs on the modern host, it cannot share semaphore with modern binary. Since this is not considered significant problem, we can avoid compat code there as well. > > > > > > Consideration 2: use the new implementation only for process-private > > > semaphores and keep using the old slow one for process-shared > > > semaphores? > > > > This is certainly safe, just requires extra work in the implementation. > > > > -- > > John Baldwin From owner-freebsd-threads@FreeBSD.ORG Wed Sep 24 15:04:36 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8146376D; Wed, 24 Sep 2014 15:04:36 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D8569E9; Wed, 24 Sep 2014 15:04:36 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id DC3F9B98A; Wed, 24 Sep 2014 11:04:34 -0400 (EDT) From: John Baldwin To: Konstantin Belousov Subject: Re: sem_post() performance Date: Wed, 24 Sep 2014 11:04:28 -0400 Message-ID: <8951456.0ca4t8PBR9@ralph.baldwin.cx> User-Agent: KMail/4.10.5 (FreeBSD/10.0-STABLE; KDE/4.10.5; amd64; ; ) In-Reply-To: <20140924144519.GL8870@kib.kiev.ua> References: <20140921213742.GA46868@stack.nl> <20140924143104.GK8870@kib.kiev.ua> <20140924144519.GL8870@kib.kiev.ua> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 24 Sep 2014 11:04:35 -0400 (EDT) Cc: adrian@freebsd.org, kib@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:04:36 -0000 On Wednesday, September 24, 2014 05:45:19 PM Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 05:31:04PM +0300, Konstantin Belousov wrote: > > On Wed, Sep 24, 2014 at 10:11:51AM -0400, John Baldwin wrote: > > > On Tuesday, September 23, 2014 11:20:00 PM Jilles Tjoelker wrote: > > > > On Mon, Sep 22, 2014 at 03:53:13PM -0400, John Baldwin wrote: > > > > > On Sunday, September 21, 2014 11:37:42 PM Jilles Tjoelker wrote: > > > > > > It has been reported that POSIX semaphores are slow, in contexts > > > > > > such as > > > > > > Python. Note that POSIX semaphores are the only synchronization > > > > > > objects > > > > > > that support use by different processes in shared memory; this > > > > > > does not > > > > > > work for mutexes and condition variables because they are pointers > > > > > > to > > > > > > the actual data structure. > > > > > > > > > > > > In fact, sem_post() unconditionally performs an umtx system call. > > > > > > > > > > *sigh* I was worried that that might be the case. > > > > > > > > > > > To avoid both lost wakeups and possible writes to a destroyed > > > > > > semaphore, > > > > > > an uncontested sem_post() must check the _has_waiters flag > > > > > > atomically > > > > > > with incrementing _count. > > > > > > > > > > > > The proper way to do this would be to take one bit from _count and > > > > > > use it for the _has_waiters flag; the definition of SEM_VALUE_MAX > > > > > > permits this. However, this would require a new set of umtx > > > > > > semaphore operations and will break ABI of process-shared > > > > > > semaphores > > > > > > (things may break if an old and a new libc access the same > > > > > > semaphore > > > > > > over shared memory). > > > > > > > > > > > > This diff only affects 32-bit aligned but 64-bit misaligned > > > > > > semaphores on 64-bit systems, and changes _count and _has_waiters > > > > > > atomically using a 64-bit atomic operation. It probably needs a > > > > > > may_alias attribute for correctness, but does not > > > > > > have > > > > > > a wrapper for that. > > > > > > > > > > It wasn't clear on first reading, but you are using aliasing to get > > > > > around the need for new umtx calls by using a 64-bit atomic op to > > > > > adjust two ints at the same time, yes? Note that since a failing > > > > > semaphore op calls into the kernel for the "hard" case, you might in > > > > > fact be able to change the ABI without breaking process-shared > > > > > semaphores. That is, suppose you left 'has_waiters' as always true > > > > > and reused the high bit of count for has_waiters. > > > > > > > > > > Would old binaries always trap into the kernel? (Not sure they > > > > > will, > > > > > especially the case where an old binary creates the semaphore, a new > > > > > binary would have to force has_waiters to true in every sem op, but > > > > > even that might not be enough.) > > > > > > > > I think that everything will break when a binary linked to old and new > > > > libcs use the same semaphore. If the new contested bit is set, the old > > > > sem_getvalue() will return garbage, the old sem_trywait() will fail > > > > even > > > > if the real count is greater than 0, the old sem_wait() and > > > > sem_timedwait() may spin if the real count is greater than 0 and the > > > > old > > > > sem_post() will fail with [EOVERFLOW]. > > > > > > Well, keep in mind you can't have a binary linked to both libc's > > > generally > > > (way too many things break if you do that, like passing the result of > > > strdup() from one libc to free() of the other, etc.). The real problem > > > case is multiple binaries. However, barring truly bizarre cases with > > > people using funky LD_LIBRARY_PATH, at least one binary would have to > > > be static to really get into trouble as in all but the bizarre cases > > > all binaries will be sharing the same libc.so.7. > > > > > > However, it was more wishful thinking on my part that we could arrange > > > things to force old binaries to still work (just slowly) while fixing > > > things cleanly.> > > > > > > I think this looks ok, but it's a shame we can't fix it more > > > > > cleanly. > > > > > > > > I don't like at all that you have to either force an exotic alignment > > > > or > > > > use unaligned atomic ops. > > > > > > > > I saw another interesting sem_ implementation in musl libc. It also > > > > uses > > > > a separate waiters word (but an int instead of a boolean), and is > > > > based > > > > on the futex calls (which we already have as wait_uint/wake umtx > > > > calls). > > > > It looks rather complicated but the musl libc people are usually good > > > > at > > > > getting this kind of stuff right. I suppose this will also break > > > > subtly > > > > when interoperating with old libc, and adding new umtx ops will > > > > simplify > > > > the algorithm. > > > > > > > > Consideration: just declare mixing process-shared semaphores with > > > > sufficiently different libc unsupported, and change SEM_MAGIC to > > > > enforce > > > > that? (This does not prevent running old binaries, as long as they're > > > > dynamically linked to libc and you use a new libc.so.) > > > > > > This only breaks old static binaries (note that we only recently got > > > pshared semaphores in 9.x or so). You could even allow partial compat > > > for those if needed by recognizing the old SEM_MAGIC when opening an > > > existing shared semaphore (so that compat would work so long as an old > > > binary created the semaphore). That compat can even be added > > > retroactively to libc if it turns out we need it. I probably think > > > that is ok, but it would be an ABI breakage that would need to be > > > documented. One thing that makes this slightly less painful is that > > > new-style semaphores in 9.0+ are per-chroot, so if you have a jail that > > > runs only old binaries, they will continue to work fine. > > > > > > I'm cc'ing Konstantin to see what he thinks. > > > > You may also get in trouble if you upgrade libc without reboot, or use > > static binary built against newer libc on older host. Yes, but we generally don't support those situations (you can do an installworld, but generally speaking you should reboot afterwards). > > But SEM_MAGIC provides at least protection against mixing incompatible > > implementations, and it seems that the checks are enough to ensure > > that both bad cases (new/old and old/new) are handled. > > > > Do we have a real use case when the old and new implementations have to > > be mixed ? Was it the same situation when the sem.c -> sem_new.c > > transition was made ? I think that the answer to the questions are > > no/yes, and we should just change magic and not worry about the issue > > until real use case appear. I think that is correct, yes. > I think it is even worse now. If the application linked against FBSD_1.0 > (ksem) semaphores implementation runs on the modern host, it cannot > share semaphore with modern binary. Correct. We changed sem_t's ABI hence the compat shims IIRC. > Since this is not considered significant problem, we can avoid compat > code there as well. I think if we leave sem_t alone there is no reason to create new compat shims for this change, but the existing FBSD_1.0 versions have to remain for people using old binaries, yes? -- John Baldwin From owner-freebsd-threads@FreeBSD.ORG Wed Sep 24 15:19:49 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C5AF184; Wed, 24 Sep 2014 15:19:49 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 70BEABA3; Wed, 24 Sep 2014 15:19:47 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8OFJhur022825 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Sep 2014 18:19:43 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8OFJhur022825 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8OFJhxq022824; Wed, 24 Sep 2014 18:19:43 +0300 (EEST) (envelope-from kostik) Date: Wed, 24 Sep 2014 18:19:43 +0300 From: Konstantin Belousov To: John Baldwin Subject: Re: sem_post() performance Message-ID: <20140924151943.GM8870@kib.kiev.ua> References: <20140921213742.GA46868@stack.nl> <20140924143104.GK8870@kib.kiev.ua> <20140924144519.GL8870@kib.kiev.ua> <8951456.0ca4t8PBR9@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8951456.0ca4t8PBR9@ralph.baldwin.cx> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: adrian@freebsd.org, kib@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:19:49 -0000 On Wed, Sep 24, 2014 at 11:04:28AM -0400, John Baldwin wrote: > On Wednesday, September 24, 2014 05:45:19 PM Konstantin Belousov wrote: > > I think it is even worse now. If the application linked against FBSD_1.0 > > (ksem) semaphores implementation runs on the modern host, it cannot > > share semaphore with modern binary. > > Correct. We changed sem_t's ABI hence the compat shims IIRC. I mean, that new and old semaphores cannot IPC. > > > Since this is not considered significant problem, we can avoid compat > > code there as well. By compat code I mean the switch on SEM_VERSION, not symversioning the libc symbols. > > I think if we leave sem_t alone there is no reason to create new compat > shims for this change, but the existing FBSD_1.0 versions have to remain for > people using old binaries, yes? Assume we applied new symver to semaphores which use new binary protocol on the existing sem_t, and, to make it reasonable, use old protocol when sem_t is accessed by older functions. Then, old binaries cannot IPC with new binaries, although both are dynamically linked to libc. IMO this is worse than the problem of different libc versions not communicating. From owner-freebsd-threads@FreeBSD.ORG Wed Sep 24 16:47:51 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B7FB57B4; Wed, 24 Sep 2014 16:47:51 +0000 (UTC) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EDDB19F2; Wed, 24 Sep 2014 16:47:50 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id a1so6094322wgh.1 for ; Wed, 24 Sep 2014 09:47:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=QsnUaocv7e6fBPeJUJXCh9cjIKK9DdE33NU12udj+ic=; b=uDOqpsj/rpN73Z5BRYDkevBEYw7rE5dJYE2ujsrZ6uFxL5B3hxdZdzasAbwj5Ibanz tIPUpeKVruqDG5n2nhsupb/xGwd3pg/hyezDnqNBzao9Wy0/qeCdK6A1kiL3Xw3FF8tI 73WCl/+BCWnqFqAohiSfaI1tm4YeO5vJLeVfV1epuRzOaC59f4ck5R9I4GasPa8jO3uG rM8NYcF5FTgjMmH94gZ9lYJoRjQuSQr1bzwiyHteggCRK5+UOYl6Bgy55cKEsuvM7wkm pxU/FCqqdZi3/J+iaYCcC2Mm2f7KtwAAb4X+pSM2uB4AwYF9SCdKYpBT5eZJOXKDBQvE mKuw== MIME-Version: 1.0 X-Received: by 10.180.9.144 with SMTP id z16mr30282349wia.26.1411577269271; Wed, 24 Sep 2014 09:47:49 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.216.106.136 with HTTP; Wed, 24 Sep 2014 09:47:49 -0700 (PDT) In-Reply-To: <20140924151943.GM8870@kib.kiev.ua> References: <20140921213742.GA46868@stack.nl> <20140924143104.GK8870@kib.kiev.ua> <20140924144519.GL8870@kib.kiev.ua> <8951456.0ca4t8PBR9@ralph.baldwin.cx> <20140924151943.GM8870@kib.kiev.ua> Date: Wed, 24 Sep 2014 09:47:49 -0700 X-Google-Sender-Auth: ZO378nbzbYwHwR5lRkkA4Ii9SGY Message-ID: Subject: Re: sem_post() performance From: Adrian Chadd To: Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Cc: Konstantin Belousov , freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 16:47:51 -0000 If we have to break things in a big way, can we at least break them in a way that makes future work somewhat backwards compatible? This is all pretty painful looking :( -a On 24 September 2014 08:19, Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 11:04:28AM -0400, John Baldwin wrote: >> On Wednesday, September 24, 2014 05:45:19 PM Konstantin Belousov wrote: >> > I think it is even worse now. If the application linked against FBSD_1.0 >> > (ksem) semaphores implementation runs on the modern host, it cannot >> > share semaphore with modern binary. >> >> Correct. We changed sem_t's ABI hence the compat shims IIRC. > I mean, that new and old semaphores cannot IPC. > >> >> > Since this is not considered significant problem, we can avoid compat >> > code there as well. > By compat code I mean the switch on SEM_VERSION, not symversioning the > libc symbols. > >> >> I think if we leave sem_t alone there is no reason to create new compat >> shims for this change, but the existing FBSD_1.0 versions have to remain for >> people using old binaries, yes? > > Assume we applied new symver to semaphores which use new binary protocol > on the existing sem_t, and, to make it reasonable, use old protocol > when sem_t is accessed by older functions. Then, old binaries cannot > IPC with new binaries, although both are dynamically linked to libc. > IMO this is worse than the problem of different libc versions not > communicating. From owner-freebsd-threads@FreeBSD.ORG Wed Sep 24 17:31:30 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DA7904F2; Wed, 24 Sep 2014 17:31:30 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B1E6AF19; Wed, 24 Sep 2014 17:31:30 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 56C7CB91A; Wed, 24 Sep 2014 13:31:29 -0400 (EDT) From: John Baldwin To: Konstantin Belousov Subject: Re: sem_post() performance Date: Wed, 24 Sep 2014 13:31:22 -0400 Message-ID: <3994706.9XWGRdTEky@ralph.baldwin.cx> User-Agent: KMail/4.10.5 (FreeBSD/10.0-STABLE; KDE/4.10.5; amd64; ; ) In-Reply-To: <20140924151943.GM8870@kib.kiev.ua> References: <20140921213742.GA46868@stack.nl> <8951456.0ca4t8PBR9@ralph.baldwin.cx> <20140924151943.GM8870@kib.kiev.ua> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 24 Sep 2014 13:31:29 -0400 (EDT) Cc: adrian@freebsd.org, kib@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 17:31:31 -0000 On Wednesday, September 24, 2014 06:19:43 PM Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 11:04:28AM -0400, John Baldwin wrote: > > On Wednesday, September 24, 2014 05:45:19 PM Konstantin Belousov wrote: > > > I think it is even worse now. If the application linked against FBSD_1.0 > > > (ksem) semaphores implementation runs on the modern host, it cannot > > > share semaphore with modern binary. > > > > Correct. We changed sem_t's ABI hence the compat shims IIRC. > > I mean, that new and old semaphores cannot IPC. Yes. > > > Since this is not considered significant problem, we can avoid compat > > > code there as well. > > By compat code I mean the switch on SEM_VERSION, not symversioning the > libc symbols. Ah, ok. My idea was we can always add that back in later if it becomes an issue (though it would only solve half the problem). > > I think if we leave sem_t alone there is no reason to create new compat > > shims for this change, but the existing FBSD_1.0 versions have to remain > > for people using old binaries, yes? > > Assume we applied new symver to semaphores which use new binary protocol > on the existing sem_t, and, to make it reasonable, use old protocol > when sem_t is accessed by older functions. Then, old binaries cannot > IPC with new binaries, although both are dynamically linked to libc. > IMO this is worse than the problem of different libc versions not > communicating. Yes, I don't think we should bump symver for this change, just change SEM_MAGIC from 'sem1' to 'sem2'. -- John Baldwin From owner-freebsd-threads@FreeBSD.ORG Fri Sep 26 15:02:24 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8732DFDB for ; Fri, 26 Sep 2014 15:02:24 +0000 (UTC) Received: from friloux.me (friloux.me [195.154.7.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C77F8CE for ; Fri, 26 Sep 2014 15:02:24 +0000 (UTC) Received: from Jetdail (143.147.84.77.rev.sfr.net [77.84.147.143]) by friloux.me (Postfix) with ESMTPSA id 1B55DFF14 for ; Fri, 26 Sep 2014 16:56:22 +0200 (CEST) Date: Fri, 26 Sep 2014 17:00:02 +0200 From: Guillaume Friloux To: freebsd-threads@freebsd.org Subject: Segfault from libthr.so. Message-ID: <20140926150001.GF7885@Jetdail> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="f5QefDQHtn8hx44O" Content-Disposition: inline User-Agent: Mutt/1.5.22 (2013-10-16) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 15:02:24 -0000 --f5QefDQHtn8hx44O Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, First of all, i am new to the BSD world. I am here to report a problem when linking an hello world with -lssp and -l= pthread. I am running : FreeBSD sds 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:= 59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 root@sds:/tmp # gcc48 main.c -o main=20 root@sds:/tmp # ./main=20 Hello World! root@sds:/tmp # gcc48 main.c -o main -lpthread root@sds:/tmp # ./main Hello World! root@sds:/tmp # gcc48 main.c -o main -lssp root@sds:/tmp # ./main Hello World! root@sds:/tmp # gcc48 main.c -o main -lssp -lpthread root@sds:/tmp # ./main Hello World! root@sds:/tmp # gcc48 main.c -o main -lpthread -lssp=20 root@sds:/tmp # ./main Segmentation fault (core dumped) root@sds:/tmp # =20 Here is what gdb says : root@sds:/tmp # gdb main main.core=20 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain condition= s. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols f= ound)... Core was generated by `main'. Program terminated with signal 11, Segmentation fault. Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libssp.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libssp.so.0 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...= done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x000000080082e604 in pthread_testcancel () from /lib/libthr.so.3 [New LWP 100071] (gdb) bt #0 0x000000080082e604 in pthread_testcancel () from /lib/libthr.so.3 #1 0x0000000800826706 in open () from /lib/libthr.so.3 #2 0x0000000800a41227 in __gets_chk () from /lib/libssp.so.0 #3 0x0000000800a413d2 in __chk_fail () from /lib/libssp.so.0 #4 0x0000000800a40ace in .init () from /lib/libssp.so.0 #5 0x00007fffffffd120 in ?? () #6 0x0000000800604691 in r_debug_state () from /libexec/ld-elf.so.1 #7 0x0000000800603d27 in __tls_get_addr () from /libexec/ld-elf.so.1 #8 0x0000000800602089 in .text () from /libexec/ld-elf.so.1 #9 0x0000000000000000 in ?? () Here is the hello world code : #include int main(int argc, char **argv) { printf("Hello World!\n"); return 0; } Thanks for taking time to read this. --f5QefDQHtn8hx44O Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUJX9xAAoJEG/Km+Gfxp5IxGIQAKGra02WfWgPHd19J5joTgnn eronppcc09LOyebEkTHHtzs9Tsvh8h6JbTwho+erBAv7AMWGFBZ41INxhK3xZ08h qD2XRn20CWML788wTpy+JQXFi3FzZGegmJdILINp5QndXhPxX1jVYUXCCmzMeH2f /fbSirT5TaEVoa9mH5XwI3LrjDKgA4Rz5w2qfIKNYLwa2UJvjbslPmDDyCZb4b37 JzyxbBhh47ERAgeWTBhqhutVJC0taEIzj9Jy3+e4yq3olx/CY6RQS7joOggvjL4M H33qUFCuYWxABDotW6BkAHMFf0dPGtgMfBoeZBw7ek7YzNj+LSiv3xanHycVzQf1 HrRiF7+qRjq1FzAfb7fwTjWVMXPYh/iP5me+6GbRw/rRa4SNw1sNQvS2a2lDPjAR 7TZWdA/ar5x9usBBWTuYviQhlbiDVINvTxgPVgBy9JXpmakLjHB2HE89zv1Nt3M+ MOXHPvfn9gcHAZ0gWMIDsP8kodxT8CPPO/Fwz844JtWW0MBSnzfkZfIZdO+C1EvC 7dj3ORRInlHs6uO7ELO0bhsjeSXSYlVwN7+g7fTb+Ki+/z2bhkcZ6sa9FQXwbsVr rj2hOFWNHwnRj20X4VsGFuXJ1gqZUm7I0QedNiuiozV/02Ez+Xr0ffXhReoiTWpR iOkNyiHY1EEcpOiDJCTj =VAh0 -----END PGP SIGNATURE----- --f5QefDQHtn8hx44O-- From owner-freebsd-threads@FreeBSD.ORG Fri Sep 26 16:43:12 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1BFB099A for ; Fri, 26 Sep 2014 16:43:12 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B1207646 for ; Fri, 26 Sep 2014 16:43:11 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8QGh6LR047149 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 26 Sep 2014 19:43:06 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8QGh6LR047149 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8QGh611047148; Fri, 26 Sep 2014 19:43:06 +0300 (EEST) (envelope-from kostik) Date: Fri, 26 Sep 2014 19:43:06 +0300 From: Konstantin Belousov To: Guillaume Friloux Subject: Re: Segfault from libthr.so. Message-ID: <20140926164306.GE8870@kib.kiev.ua> References: <20140926150001.GF7885@Jetdail> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140926150001.GF7885@Jetdail> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 16:43:12 -0000 On Fri, Sep 26, 2014 at 05:00:02PM +0200, Guillaume Friloux wrote: > Hello, > > First of all, i am new to the BSD world. > > I am here to report a problem when linking an hello world with -lssp and -lpthread. > I am running : > FreeBSD sds 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > > root@sds:/tmp # gcc48 main.c -o main > root@sds:/tmp # ./main > Hello World! > root@sds:/tmp # gcc48 main.c -o main -lpthread > root@sds:/tmp # ./main > Hello World! > root@sds:/tmp # gcc48 main.c -o main -lssp > root@sds:/tmp # ./main > Hello World! > root@sds:/tmp # gcc48 main.c -o main -lssp -lpthread > root@sds:/tmp # ./main > Hello World! > root@sds:/tmp # gcc48 main.c -o main -lpthread -lssp > root@sds:/tmp # ./main > Segmentation fault (core dumped) > root@sds:/tmp # > > Here is what gdb says : > root@sds:/tmp # gdb main main.core > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... > Core was generated by `main'. > Program terminated with signal 11, Segmentation fault. > Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done. > Loaded symbols for /lib/libthr.so.3 > Reading symbols from /lib/libssp.so.0...(no debugging symbols found)...done. > Loaded symbols for /lib/libssp.so.0 > Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. > Loaded symbols for /lib/libc.so.7 > Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. > Loaded symbols for /libexec/ld-elf.so.1 > #0 0x000000080082e604 in pthread_testcancel () from /lib/libthr.so.3 > [New LWP 100071] > (gdb) bt > #0 0x000000080082e604 in pthread_testcancel () from /lib/libthr.so.3 > #1 0x0000000800826706 in open () from /lib/libthr.so.3 > #2 0x0000000800a41227 in __gets_chk () from /lib/libssp.so.0 > #3 0x0000000800a413d2 in __chk_fail () from /lib/libssp.so.0 > #4 0x0000000800a40ace in .init () from /lib/libssp.so.0 > #5 0x00007fffffffd120 in ?? () > #6 0x0000000800604691 in r_debug_state () from /libexec/ld-elf.so.1 > #7 0x0000000800603d27 in __tls_get_addr () from /libexec/ld-elf.so.1 > #8 0x0000000800602089 in .text () from /libexec/ld-elf.so.1 > #9 0x0000000000000000 in ?? () > > Here is the hello world code : > #include > > int main(int argc, char **argv) > { > printf("Hello World!\n"); > return 0; > } > > > Thanks for taking time to read this. So, what is the problem, from your PoV ? Linking libssp after libthr causes libssp constructors run before libthr initialization, which causes the behaviour you see. The libssp uses something which is interposed by libthr, but still not ready at the libssp init time. Your example of linking with different order demostrates you the right way to treat libthr. So again, what is the problem ? From owner-freebsd-threads@FreeBSD.ORG Fri Sep 26 19:37:49 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AA95FDDE for ; Fri, 26 Sep 2014 19:37:49 +0000 (UTC) Received: from friloux.me (friloux.me [195.154.7.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 70DB2B16 for ; Fri, 26 Sep 2014 19:37:49 +0000 (UTC) Received: from Jetdail (143.147.84.77.rev.sfr.net [77.84.147.143]) by friloux.me (Postfix) with ESMTPSA id 1B944FF14; Fri, 26 Sep 2014 21:37:21 +0200 (CEST) Date: Fri, 26 Sep 2014 21:41:01 +0200 From: Guillaume Friloux To: Konstantin Belousov Subject: Re: Segfault from libthr.so. Message-ID: <20140926194100.GG7885@Jetdail> References: <20140926150001.GF7885@Jetdail> <20140926164306.GE8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="h3LYUU6HlUDSAOzy" Content-Disposition: inline In-Reply-To: <20140926164306.GE8870@kib.kiev.ua> User-Agent: Mutt/1.5.22 (2013-10-16) Cc: freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 19:37:49 -0000 --h3LYUU6HlUDSAOzy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 14/09/26, Konstantin Belousov wrote: > So, what is the problem, from your PoV ? > Linking libssp after libthr causes libssp constructors run before > libthr initialization, which causes the behaviour you see. The libssp > uses something which is interposed by libthr, but still not ready at > the libssp init time. Your example of linking with different order > demostrates you the right way to treat libthr. >=20 > So again, what is the problem ? Hello, to me this behavior did not look normal. I did not look at the code of both libs, i asked for help on IRC and got se= nt here. What surprised me is that i have never met this on GNU/Linux (i repeat, i a= m new to BSD). Isn't there a way to sanitize this and not jump into a segfault ? PS : I might just have been trolled by an IRC guy. --h3LYUU6HlUDSAOzy Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUJcFMAAoJEG/Km+Gfxp5IX1EP/jyV4UQ539qLp8tiQzfnOz30 5IAtQ41O8/LC8g9nyAi1otwsbOlpLohnpUxFN2fgaaSjwI/v8c7sNfw4Xm6TkzXl /NbNJfmcgqTJ4Xcq4U6zr8oEn2uBrQ632MK3OtyBLmsrfiXqXOs5QJ1P4BCgVFPp y4QGQUcsGIZ2/0SEtqBty+5gUPA0OwwOeiC2Q9wn5PVsn/ywbo7DIhr5yhtJzYm+ BKstvT4mxdVgXFT92jk6qLYyzb44lrazK8M9ezs5b+aaccMr0yESIyRkiSNcpCXd dNv46CYJ2J+OyebwXJqeAUJVwqNj+f8CDVcPOUumVsVGNW+xYKbc8jpaMyGlV5K/ wNSOAGWTELj+53TKnUoJCsEZxbp70E0FjZK9JuGxZ9+hcXCqMBqKVlaNNvmivdE7 b7w8NhgCDsq97qEuCa/ggvO2JhMvHoeogCH+rX7FonOWdSB3JvcSYZZHzlzI8lxQ GxMiI80cmMj8+Hir36liLNRIZV0qENkUty0EkFA8BSlxToNWxEsXQk3Eu6wnLGJh 4pLfm8aBCQYJFOPBAf4v8w+df0AiovBYA0YkMqS5lbH7+bfbUL7rN02IBX+Wlq2A KkB7knDZ3ZGsnI8JeYXt+RtuM+KRIbZ/pfCs4jOkL7TTSBTR/AuW9OyCNotgMLHn XKrHyTKN44RygA4GuCdf =MtJy -----END PGP SIGNATURE----- --h3LYUU6HlUDSAOzy-- From owner-freebsd-threads@FreeBSD.ORG Sat Sep 27 06:52:44 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8F6C02B2 for ; Sat, 27 Sep 2014 06:52:44 +0000 (UTC) Received: from friloux.me (friloux.me [195.154.7.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 55EC6762 for ; Sat, 27 Sep 2014 06:52:43 +0000 (UTC) Received: from Jetdail (143.147.84.77.rev.sfr.net [77.84.147.143]) by friloux.me (Postfix) with ESMTPSA id 00F8AFF59; Sat, 27 Sep 2014 08:52:15 +0200 (CEST) Date: Sat, 27 Sep 2014 08:55:56 +0200 From: Guillaume Friloux To: Konstantin Belousov Subject: Re: Segfault from libthr.so. Message-ID: <20140927065556.GH7885@Jetdail> References: <20140926150001.GF7885@Jetdail> <20140926164306.GE8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MPkR1dXiUZqK+927" Content-Disposition: inline In-Reply-To: <20140926164306.GE8870@kib.kiev.ua> User-Agent: Mutt/1.5.22 (2013-10-16) Cc: freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 06:52:44 -0000 --MPkR1dXiUZqK+927 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 14/09/26, Konstantin Belousov wrote: > So, what is the problem, from your PoV ? > Linking libssp after libthr causes libssp constructors run before > libthr initialization, which causes the behaviour you see. The libssp > uses something which is interposed by libthr, but still not ready at > the libssp init time. Your example of linking with different order > demostrates you the right way to treat libthr. >=20 > So again, what is the problem ? Sorry to disturb you again, but i would have more access to your knowledge. I understand what you say, but in my case, it disturbs me a little. Lets take this project as example : https://github.com/gfriloux/botman As you can see, it uses an m4 macro that will force link to ssp (configure.= ac:38). Later on, deps will be declared (configure.ac:87). One of those deps is ein= a. Eina will declare -lpthread from its .pc file. And this is how i get to the state of having -lssp -lpthread. In my app, i dont directly start any thread, lower libs do. So it doesnt se= em logical to add -lpthread before declaring the deps. So should i just remove this m4 macro that seems to be use on quite some pr= ojects, so it works on BSD ? What is the best way to do it, in your opinion ? --MPkR1dXiUZqK+927 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUJl98AAoJEG/Km+Gfxp5IQA4P/A+Baryv14J9a0+aYIR81Gum xiTo+3NzLky/tLmVFXRaF0X5D01Jvy3EwqwdToVRQqNbJFYbKQgrFKnTnAcA7IfL 3W3vpUabiJ8gPoy72ZpiTKsC2UgJo/lZ+t3TNE6mnzkbhVQU3Uh08IAA0md0zwS1 M9sCENdca4PVxSlOTAaLpFcpgasDJVZj3PF/hnwoNSldx9lRt8wPm7yd3LbLFPEm gdLsZcBJcM9mHhmGC+4GaxsDDbz5SZ9py7rqYYbba2+WdPztyiUEb8db6fsFU6+e Fny60oi6akFadcRgPhZ6dXbwEc0bIeklFGB9OMY3ReuQ8Fda/D5q5MNeNPkyin2M 5/L8d/md19yOJne6tdZi0o8cyx10UflwEJVZnkRLmxHp1f9ec9d7/s1KHeBRKOhg 0i6enl2YnEK1klVK557N6BNP2zURC4QQGrvh3vtLMk7Kj0G3S6r76nrVsIy1Mvk1 IzOwb7qT9JT6bPkPBEOhMz0iZMQovUTY2RFYpzysDCC0k3i0rBjnEHKRYOqMqQOV 5Gx8quRzifTFAlAJgDNqowpdDc6wkd3awOVdsK2RkkqNsKryTOEAX+TpyxhRGVxQ yOO+pl3EY9w4wWJijAH69Vjt1hDqPggaZa+qHwu5KP0WVy9fTPjK+QprvXQDpcLH 0KPWsyP2hLH17pao/nMc =RSSZ -----END PGP SIGNATURE----- --MPkR1dXiUZqK+927-- From owner-freebsd-threads@FreeBSD.ORG Sat Sep 27 08:36:28 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 304089D4 for ; Sat, 27 Sep 2014 08:36:28 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 97022F68 for ; Sat, 27 Sep 2014 08:36:27 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8R8aLZq054030 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 27 Sep 2014 11:36:21 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8R8aLZq054030 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8R8aL0b054029; Sat, 27 Sep 2014 11:36:21 +0300 (EEST) (envelope-from kostik) Date: Sat, 27 Sep 2014 11:36:21 +0300 From: Konstantin Belousov To: Guillaume Friloux Subject: Re: Segfault from libthr.so. Message-ID: <20140927083621.GJ8870@kib.kiev.ua> References: <20140926150001.GF7885@Jetdail> <20140926164306.GE8870@kib.kiev.ua> <20140927065556.GH7885@Jetdail> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140927065556.GH7885@Jetdail> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 08:36:28 -0000 On Sat, Sep 27, 2014 at 08:55:56AM +0200, Guillaume Friloux wrote: > On 14/09/26, Konstantin Belousov wrote: > > So, what is the problem, from your PoV ? > > Linking libssp after libthr causes libssp constructors run before > > libthr initialization, which causes the behaviour you see. The libssp > > uses something which is interposed by libthr, but still not ready at > > the libssp init time. Your example of linking with different order > > demostrates you the right way to treat libthr. > > > > So again, what is the problem ? > > Sorry to disturb you again, but i would have more access to your knowledge. > I understand what you say, but in my case, it disturbs me a little. > > Lets take this project as example : https://github.com/gfriloux/botman > > As you can see, it uses an m4 macro that will force link to ssp (configure.ac:38). > Later on, deps will be declared (configure.ac:87). One of those deps is eina. > Eina will declare -lpthread from its .pc file. > > And this is how i get to the state of having -lssp -lpthread. So this is right order. > In my app, i dont directly start any thread, lower libs do. So it doesnt seem > logical to add -lpthread before declaring the deps. > So should i just remove this m4 macro that seems to be use on quite some projects, > so it works on BSD ? > > What is the best way to do it, in your opinion ? Just put -lpthread last in the command line (automake variable) for linker. This should work portable on all systems I know. From owner-freebsd-threads@FreeBSD.ORG Sat Sep 27 21:49:47 2014 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A1A36E6 for ; Sat, 27 Sep 2014 21:49:47 +0000 (UTC) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5C91E768 for ; Sat, 27 Sep 2014 21:49:46 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1XXzrz-0000tb-2H for freebsd-threads@freebsd.org; Sat, 27 Sep 2014 23:49:35 +0200 Received: from a91-154-115-217.elisa-laajakaista.fi ([91.154.115.217]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 27 Sep 2014 23:49:35 +0200 Received: from rakuco by a91-154-115-217.elisa-laajakaista.fi with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Sat, 27 Sep 2014 23:49:35 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-threads@freebsd.org From: Raphael Kubo da Costa Subject: Re: Segfault from libthr.so. Date: Sun, 28 Sep 2014 00:49:21 +0300 Lines: 15 Message-ID: <86ppegg2e6.fsf@FreeBSD.org> References: <20140926150001.GF7885@Jetdail> <20140926164306.GE8870@kib.kiev.ua> <20140927065556.GH7885@Jetdail> <20140927083621.GJ8870@kib.kiev.ua> Mime-Version: 1.0 Content-Type: text/plain X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: a91-154-115-217.elisa-laajakaista.fi User-Agent: Gnus/5.130012 (Ma Gnus v0.12) Emacs/24.3 (berkeley-unix) Cancel-Lock: sha1:3MxJG3BOu3N6LKisXAdHIkcTTdk= X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 21:49:47 -0000 Konstantin Belousov writes: >> In my app, i dont directly start any thread, lower libs do. So it >> doesnt seem logical to add -lpthread before declaring the deps. So >> should i just remove this m4 macro that seems to be use on quite some >> projects, so it works on BSD ? >> >> What is the best way to do it, in your opinion ? > > Just put -lpthread last in the command line (automake variable) for linker. > This should work portable on all systems I know. Alternatively, you could pass -pthread instead of -lpthread: the compiler should then take care of passing -lpthread in the right position -- it's what we use in the ports tree, for example.