From owner-svn-src-head@freebsd.org Mon Feb 4 07:00:25 2019 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CCAEB14C1256; Mon, 4 Feb 2019 07:00:25 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3766E70C6B; Mon, 4 Feb 2019 07:00:25 +0000 (UTC) (envelope-from kib@freebsd.org) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x1470Hmt000889 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 4 Feb 2019 09:00:20 +0200 (EET) (envelope-from kib@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x1470Hmt000889 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x1470HTU000888; Mon, 4 Feb 2019 09:00:17 +0200 (EET) (envelope-from kib@freebsd.org) X-Authentication-Warning: tom.home: kostik set sender to kib@freebsd.org using -f Date: Mon, 4 Feb 2019 09:00:17 +0200 From: Konstantin Belousov To: mmel@freebsd.org Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r343566 - in head/lib/libthr: . thread Message-ID: <20190204070017.GO24863@kib.kiev.ua> References: <201901292246.x0TMkjQH074121@repo.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.2 (2019-01-07) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Feb 2019 07:00:26 -0000 On Mon, Feb 04, 2019 at 07:31:19AM +0100, Michal Meloun wrote: > On 29.01.2019 23:46, Konstantin Belousov wrote: > > Author: kib > > Date: Tue Jan 29 22:46:44 2019 > > New Revision: 343566 > > URL: https://svnweb.freebsd.org/changeset/base/343566 > > > > Log: > > Untangle jemalloc and mutexes initialization. > > > > The need to use libc malloc(3) from some places in libthr always > > caused issues. For instance, per-thread key allocation was switched to > > use plain mmap(2) to get storage, because some third party mallocs > > used keys for implementation of calloc(3). > > > > Even more important, libthr calls calloc(3) during initialization of > > pthread mutexes, and jemalloc uses pthread mutexes. Jemalloc provides > > some way to both postpone the initialization, and to make > > initialization to use specialized allocator, but this is very fragile > > and often breaks. See the referenced PR for another example. > > > > Add the small malloc implementation used by rtld, to libthr. Use it in > > thr_spec.c and for mutexes initialization. This avoids the issues with > > mutual dependencies between malloc and libthr in principle. The > > drawback is that some more allocations are not interceptable for > > alternate malloc implementations. There should be not too much memory > > use from this allocator, and the alternative, direct use of mmap(2) is > > obviously worse. > > > > PR: 235211 > > MFC after: 2 weeks > > Sponsored by: The FreeBSD Foundation > > Differential revision: https://reviews.freebsd.org/D18988 > > > This broke ARM static binaries (at least rescue/rescue). From first > look it seems that __pthread_mutex_init() invoked by atomic_init() is > called before curthread is set (before _thread_init_hack()). > > > root@tegra124:/usr/src # gdb --args > /usr/obj/usr/src/arm.armv7/rescue/rescue/rescue sh > ... > Reading symbols from /usr/obj/usr/src/arm.armv7/rescue/rescue/rescue...done. > (gdb) b _thread_init_hack > Breakpoint 1 at 0x67cad0: file /usr/src/lib/libthr/thread/thr_init.c, > line 296. > (gdb) r > Starting program: /usr/obj/usr/src/arm.armv7/rescue/rescue/rescue sh > > Program received signal SIGSEGV, Segmentation fault. > __thr_calloc (num=1, size=) at > /usr/src/lib/libthr/thread/thr_malloc.c:82 > 82 thr_malloc_lock(curthread); > (gdb) bt > #0 __thr_calloc (num=1, size=) at > /usr/src/lib/libthr/thread/thr_malloc.c:82 > #1 0x00676e1c in mutex_init (mutex=, > mutex_attr=, calloc_cb=) > at /usr/src/lib/libthr/thread/thr_mutex.c:294 > #2 __pthread_mutex_init (mutex=0xc6e948 , > mutex_attr=) at /usr/src/lib/libthr/thread/thr_mutex.c:393 > #3 0x001d41f0 in handle_static_init (argc=2, argv=, > env=) at /usr/src/lib/csu/common/ignore_init.c:124 > #4 0x001d40e8 in __start (argc=2, argv=, env= out>, ps_strings=, obj=0x0, cleanup=0x0) > at /usr/src/lib/csu/arm/crt1.c:112 > #5 0x001d4000 in ?? () > Backtrace stopped: previous frame identical to this frame (corrupt stack?) > (gdb) f 3 > #3 0x001d41f0 in handle_static_init (argc=2, argv=, > env=) at /usr/src/lib/csu/common/ignore_init.c:124 > 124 fn(argc, argv, env); > (gdb) list > 119 _init(); > 120 array_size = __init_array_end - __init_array_start; > 121 for (n = 0; n < array_size; n++) { > 122 fn = __init_array_start[n]; > 123 if ((uintptr_t)fn != 0 && (uintptr_t)fn != 1) > 124 fn(argc, argv, env); > 125 } > 126 } > (gdb) p n > $1 = 20 > (gdb) p fn > $6 = (void (*)(int, char **, char **)) 0x63f21c > (gdb) p (void *)&__init_array_start > $19 = (void *) 0xa7ab60 > (gdb) p (void *)&__init_array_end > $20 = (void *) 0xa7abc0 > (gdb) x/24a &__init_array_start > 0xa7ab60: 0x1d4270 > 0x2d926c <_$$hide$$ ifconfig.lo ifconfig_ctor> > 0x2d98e0 <_$$hide$$ ifconfig.lo link_ctor> > 0x2d9cb4 <_$$hide$$ ifconfig.lo inet_ctor> > 0xa7ab70: 0x2da2e0 <_$$hide$$ ifconfig.lo inet6_ctor> > 0x2db790 <_$$hide$$ ifconfig.lo clone_ctor> > 0x2dbb8c <_$$hide$$ ifconfig.lo mac_ctor> > 0x2dbe70 <_$$hide$$ ifconfig.lo ifmedia_ctor> > 0xa7ab80: 0x2dced0 <_$$hide$$ ifconfig.lo fib_ctor> > 0x2dd118 <_$$hide$$ ifconfig.lo vlan_ctor> > 0x2dd668 <_$$hide$$ ifconfig.lo vxlan_ctor> > 0x2df45c <_$$hide$$ ifconfig.lo gre_ctor> > 0xa7ab90: 0x2df6d8 <_$$hide$$ ifconfig.lo gif_ctor> > 0x2df874 <_$$hide$$ ifconfig.lo ipsec_ctor> > 0x2e2144 <_$$hide$$ ifconfig.lo ieee80211_ctor> > 0x2f0a10 <_$$hide$$ ifconfig.lo carp_ctor> > 0xa7aba0: 0x2f0e6c <_$$hide$$ ifconfig.lo group_ctor> > 0x2f199c <_$$hide$$ ifconfig.lo pfsync_ctor> > 0x2f1a04 <_$$hide$$ ifconfig.lo bridge_ctor> > 0x2f35dc <_$$hide$$ ifconfig.lo lagg_ctor> > 0xa7abb0: 0x63f21c > 0x67cad0 <_thread_init_hack> > 0x90e5b4 > 0x9a6b98 > > So it's clear that order of static constructors is invalid (moreover, I > thing that we are inside undefined area here). > Any idea/hint how to fix this. Try the following (I did not even compiled). It might require additional handling of NULL curthread in thr_malloc.c, in which case the locking should be elided. diff --git a/lib/libthr/thread/thr_malloc.c b/lib/libthr/thread/thr_malloc.c index 157c72f10d6..8b72a1840f7 100644 --- a/lib/libthr/thread/thr_malloc.c +++ b/lib/libthr/thread/thr_malloc.c @@ -46,6 +46,8 @@ void __thr_malloc_init(void) { + if (npagesizes != 0) + return; npagesizes = getpagesizes(pagesizes_d, nitems(pagesizes_d)); if (npagesizes == -1) { npagesizes = 1; diff --git a/lib/libthr/thread/thr_mutex.c b/lib/libthr/thread/thr_mutex.c index f6f37c1264e..4db65384331 100644 --- a/lib/libthr/thread/thr_mutex.c +++ b/lib/libthr/thread/thr_mutex.c @@ -390,6 +390,7 @@ __pthread_mutex_init(pthread_mutex_t * __restrict mutex, } if (mutex_attr == NULL || (*mutex_attr)->m_pshared == PTHREAD_PROCESS_PRIVATE) { + __thr_malloc_init(); return (mutex_init(mutex, mutex_attr ? *mutex_attr : NULL, __thr_calloc)); }