From owner-freebsd-threads@FreeBSD.ORG Mon Mar 9 14:00:04 2015 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 71BD978B for ; Mon, 9 Mar 2015 14:00:04 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3EE53C5A for ; Mon, 9 Mar 2015 14:00:04 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t29E04CB059251 for ; Mon, 9 Mar 2015 14:00:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-threads@FreeBSD.org Subject: [Bug 198452] libthr/rtld deadlock Date: Mon, 09 Mar 2015 14:00:04 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: threads X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dpejesh@yahoo.com X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-threads@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2015 14:00:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198452 Bug ID: 198452 Summary: libthr/rtld deadlock Product: Base System Version: 11.0-CURRENT Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: threads Assignee: freebsd-threads@FreeBSD.org Reporter: dpejesh@yahoo.com I've run into what appears to be a deadlock with the following code when linked against libthr. Prior to base r245630 this worked as intended. Reverting base r245630 and base r249425 in HEAD appears to fix it. #include #include int32_t callback(struct dl_phdr_info *info, size_t size, void *data) { dlopen(info->dlpi_name, RTLD_LAZY); return 0; } int main() { dl_iterate_phdr(callback, 0); return 0; } When linked with libthr the program deadlocks on the call to dlopen(). clang -g -lthr -o main main.c The backtrace I'm seeing is: (gdb) bt #0 0x0000000800833ddc in _umtx_op_err () from /lib/libthr.so.3 #1 0x000000080082a29f in __thr_rwlock_wrlock (rwlock=0x800a42900, tsp=) at /usr/src/lib/libthr/thread/thr_umtx.c:296 #2 0x00000008008317f3 in _thr_rwlock_wrlock (rwlock=, tsp=) at /usr/src/lib/libthr/thread/thr_umtx.h:204 #3 _thr_rtld_wlock_acquire (lock=0x800a42900) at /usr/src/lib/libthr/thread/thr_rtld.c:141 #4 0x000000080060b5c2 in wlock_acquire (lock=0x80081f8a0 , lockstate=0x7fffffffe058) at /usr/src/libexec/rtld-elf/rtld_lock.c:213 #5 0x000000080060905f in dlopen_object () from /libexec/ld-elf.so.1 #6 0x00000008006060a3 in rtld_dlopen (name=0x7fffffffefcc "/root/deadlock/main", fd=-1, mode=) at /usr/src/libexec/rtld-elf/rtld.c:2955 #7 0x000000000040081f in callback (info=0x7fffffffeb08, size=64, data=0x0) at main.c:7 #8 0x0000000800606eeb in dl_iterate_phdr (callback=0x4007e0 , param=0x0) at /usr/src/libexec/rtld-elf/rtld.c:3416 #9 0x0000000000400858 in main () at main.c:12 -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-threads@FreeBSD.ORG Mon Mar 9 15:07:04 2015 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 93ECA1AB for ; Mon, 9 Mar 2015 15:07:04 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7A1E46B1 for ; Mon, 9 Mar 2015 15:07:04 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t29F74d5053314 for ; Mon, 9 Mar 2015 15:07:04 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-threads@FreeBSD.org Subject: [Bug 198452] libthr/rtld deadlock Date: Mon, 09 Mar 2015 15:07:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: threads X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-threads@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2015 15:07:04 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198452 Konstantin Belousov changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |Works As Intended CC| |kib@FreeBSD.org Status|New |Closed --- Comment #1 from Konstantin Belousov --- (In reply to dpejesh from comment #0) Dlopen/dlclose cannot work in the context of the dl_iterate_phdr(3) callback. At least, because the dl_iterate_phdr() walks over the list of the loaded shared objects, so in the language terms, it is the iterator over collection, and collection must be stable. The callback API does not suppose a possibitily of restart. That said, callback is intended to perform some lookups on the per object basis, e.g. enumerate exception tables, debug sections and like. The revisions you point out as causing the problem obviously mean that this never worked anyway. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-threads@FreeBSD.ORG Mon Mar 9 18:10:14 2015 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3285CA4B for ; Mon, 9 Mar 2015 18:10:14 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 19666FBC for ; Mon, 9 Mar 2015 18:10:14 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t29IADKT076762 for ; Mon, 9 Mar 2015 18:10:13 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-threads@FreeBSD.org Subject: [Bug 198452] libthr/rtld deadlock Date: Mon, 09 Mar 2015 18:10:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: threads X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dpejesh@yahoo.com X-Bugzilla-Status: Closed X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-threads@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2015 18:10:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198452 --- Comment #2 from dpejesh@yahoo.com --- > Dlopen/dlclose cannot work in the context of the dl_iterate_phdr(3) callback. > At least, because the dl_iterate_phdr() walks over the list of the loaded > shared objects, so in the language terms, it is the iterator over collection, > and collection must be stable. The callback API does not suppose a > possibitily of restart. That makes sense, but this use to work pre-10.0 and currently works on Linux. It shouldn't be difficult to work around, but the new behavior seems like a regression which is why I opened the ticket. I can understand dlopen() failing if it has to load a new shared object and update its internal list, but if it's already loaded all it should need to do is update the counters and still return the handle, right? > That said, callback is intended to perform some lookups on the per object > basis, e.g. enumerate exception tables, debug sections and like. > The revisions you point out as causing the problem obviously mean that this > never worked anyway. Not sure why you're saying that. Before r245630 it worked, and reverting that change on HEAD fixes it too. It also works when not linked to libthr both before and after r245630. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-threads@FreeBSD.ORG Mon Mar 9 19:00:25 2015 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4503F752 for ; Mon, 9 Mar 2015 19:00:25 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2BACF8AE for ; Mon, 9 Mar 2015 19:00:25 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id t29J0P6N096833 for ; Mon, 9 Mar 2015 19:00:25 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-threads@FreeBSD.org Subject: [Bug 198452] libthr/rtld deadlock Date: Mon, 09 Mar 2015 19:00:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: threads X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kib@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-threads@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Mar 2015 19:00:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=198452 --- Comment #3 from Konstantin Belousov --- (In reply to dpejesh from comment #2) Technically, what happens is that dl_iterate_phdr() locks the rtld_bind_lock, and dlopen() needs the same lock. This is reasonable, since the lock protects the structures which are used by iterator and modified by dlopen(), just for example, the list of the loaded objects. Generally, we cannot upgrade read-lock to write, since there may be other read lock owners. We cannot drop read lock, to wait for write, since this invalidates previous iterations. I said that your reference to the commits is strange, because this is definitely not the commits you point out that introduces the behavior. It is there from the moment when locks were added to dl_iterate_phdr(). You are tripping on the stated revisions since that revs force to use real locking when libthr.so is loaded into the process. The single-threaded processes use some sort of fake locking in rtld. I.e. the 'bug' is there for long time. WRT Linux, their dl_iterate_phdr() is much less safe, so to say, then ours. For instance, their dl_iterate_phdr() should not be used from the signal handler context, which causes trouble for libunwind, and is one of the reasons why I do not want to change the code to e.g. dropping lock during the calls to callback and using sentinel to remember the position to iterate. -- You are receiving this mail because: You are the assignee for the bug.