From owner-freebsd-current@freebsd.org Sun Mar 20 07:28:29 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A0F4AD6F74 for ; Sun, 20 Mar 2016 07:28:29 +0000 (UTC) (envelope-from oleg@opentransfer.com) Received: from oleg.opentransfer.com (oleg.opentransfer.com [91.217.144.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "oleg-10.opentransfer.com", Issuer "oleg-10.opentransfer.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BD7501A47 for ; Sun, 20 Mar 2016 07:28:28 +0000 (UTC) (envelope-from oleg@opentransfer.com) Received: from asus.theweb.org.ua ([10.0.8.4]) by oleg.opentransfer.com (8.15.2/8.15.2) with ESMTPS id u2K7VkeH038369 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Sun, 20 Mar 2016 09:31:48 +0200 (EET) (envelope-from oleg@opentransfer.com) Received: from asus.theweb.org.ua (localhost [127.0.0.1]) by asus.theweb.org.ua (8.15.2/8.15.2) with ESMTPS id u2K7SEpd002193 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 20 Mar 2016 09:28:14 +0200 (EET) (envelope-from oleg@opentransfer.com) Received: (from oleg@localhost) by asus.theweb.org.ua (8.15.2/8.15.2/Submit) id u2K7SDOU002192; Sun, 20 Mar 2016 09:28:13 +0200 (EET) (envelope-from oleg@opentransfer.com) X-Authentication-Warning: asus.theweb.org.ua: oleg set sender to oleg@opentransfer.com using -f From: "Oleg V. Nauman" To: Konstantin Belousov Cc: freebsd-current@freebsd.org Subject: Re: Fatal error 'mutex is on list' at line 139 in file /usr/src/lib/libthr/thread/thr_mutex.c (errno = 35) Date: Sun, 20 Mar 2016 09:28:13 +0200 Message-ID: <9991609.BzAFArhlj3@asus.theweb.org.ua> Organization: Ecommerce LLC User-Agent: KMail/4.14.3 (FreeBSD/11.0-CURRENT; KDE/4.14.3; amd64; ; ) In-Reply-To: <20160319194757.GE1741@kib.kiev.ua> References: <5093647.qxI0C33PyG@asus.theweb.org.ua> <5286161.5Mbx8epR8j@asus.theweb.org.ua> <20160319194757.GE1741@kib.kiev.ua> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Mar 2016 07:28:29 -0000 On Saturday 19 March 2016 21:47:57 Konstantin Belousov wrote: > On Sat, Mar 19, 2016 at 04:03:06PM +0200, Oleg V. Nauman wrote: > > Core was generated by `akonadi_baloo_index'. > > Program terminated with signal SIGABRT, Aborted. > > #0 0x0000000805630d6a in thr_kill () from /lib/libc.so.7 > > (gdb) bt > > #0 0x0000000805630d6a in thr_kill () from /lib/libc.so.7 > > #1 0x0000000805630d3b in __raise (s=6) at > > /usr/src/lib/libc/gen/raise.c:52 > > #2 0x0000000805630ca9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65 > > #3 0x00000008053564b4 in _thread_exit ( > > > > fname=0x805357b70 "/usr/src/lib/libthr/thread/thr_mutex.c", > > lineno=139, > > msg=0x805357b97 "mutex is on list") at > > > > /usr/src/lib/libthr/thread/thr_exit.c:182 > > #4 0x000000080534cddc in mutex_assert_not_owned (m=0x80064d000) > > > > at /usr/src/lib/libthr/thread/thr_mutex.c:139 > > > > #5 0x000000080534cfb9 in enqueue_mutex (curthread=0x80e015000, > > m=0x80064d000)> > > at /usr/src/lib/libthr/thread/thr_mutex.c:383 > > > > #6 0x000000080534d213 in mutex_lock_common (m=0x80064d000, > > abstime=0x7fffffffd4e8, > > > > cvattach=0) at /usr/src/lib/libthr/thread/thr_mutex.c:533 > > > > #7 0x000000080534c6be in __pthread_mutex_timedlock (mutex=0x811c00008, > > > > abstime=0x7fffffffd4e8) at /usr/src/lib/libthr/thread/thr_mutex.c:566 > > > > (gdb) f 7 > > #7 0x000000080534c6be in __pthread_mutex_timedlock (mutex=0x811c00008, > > > > abstime=0x7fffffffd4e8) at /usr/src/lib/libthr/thread/thr_mutex.c:566 > > > > 566 ret = mutex_lock_common(m, abstime, 0); > > (gdb) p *mutex > > $1 = (pthread_mutex_t) 0x8000000000000001 > > (gdb) p m > > $2 = (struct pthread_mutex *) 0x80064d000 > > (gdb) p *m > > $3 = {m_lock = {m_owner = -2147383372, m_flags = 1, m_ceilings = {0, 0}, > > m_spare = {0, 0, 0, > > > > 0}}, m_flags = 1, m_owner = 100276, m_count = 0, m_spinloops = 0, > > > > m_yieldloops = 0, > > > > m_qe = {tqe_next = 0x0, tqe_prev = 0x0}, m_pqe = {tqe_next = 0x0, > > tqe_prev => > > 0x0}} > > (gdb) p *curthread > > No symbol "curthread" in current context. > > (gdb) > > curthread is available e.q. in the frame 5. > > The content from the printout is reasonable, but now it contradicts to the > assertion fired, since both checked pointers are NULL, as reported by gdb. > > Please add the following debugging patch on top of the previous change > and reproduce the issue. I need the same info, but please also provide > exact gasp message from libthr, which is enchanced in the patch below. > > diff --git a/lib/libthr/thread/thr_mutex.c b/lib/libthr/thread/thr_mutex.c > index 30a8be2..3342c9f 100644 > --- a/lib/libthr/thread/thr_mutex.c > +++ b/lib/libthr/thread/thr_mutex.c > @@ -124,8 +124,14 @@ mutex_assert_is_owned(struct pthread_mutex *m) > { > > #if defined(_PTHREADS_INVARIANTS) > - if (__predict_false(m->m_qe.tqe_prev == NULL)) > - PANIC("mutex is not on list"); > + if (__predict_false(m->m_qe.tqe_prev == NULL)) { > + char msg[128]; > + snprintf(msg, sizeof(msg), > + "mutex %p own %#x %#x is not on list %p %p", > + m, m->m_lock.m_owner, m->m_owner, m->m_qe.tqe_prev, > + m->m_qe.tqe_next); > + PANIC(msg); > + } > #endif > } > > @@ -135,8 +141,14 @@ mutex_assert_not_owned(struct pthread_mutex *m) > > #if defined(_PTHREADS_INVARIANTS) > if (__predict_false(m->m_qe.tqe_prev != NULL || > - m->m_qe.tqe_next != NULL)) > - PANIC("mutex is on list"); > + m->m_qe.tqe_next != NULL)) { > + char msg[128]; > + snprintf(msg, sizeof(msg), > + "mutex %p own %#x %#x is on list %p %p", > + m, m->m_lock.m_owner, m->m_owner, m->m_qe.tqe_prev, > + m->m_qe.tqe_next); > + PANIC(msg); > + } > #endif > } Fatal error 'mutex 0x800632000 own 0x1885c 0x1885c is on list 0x0 0x80d46ebc0' at line 155 in file /usr/src/lib/libthr/thread/thr_mutex.c (errno = 2 ) What I have got after applying this patch: #0 0x0000000805913d6a in thr_kill () from /lib/libc.so.7 #1 0x0000000805913d3b in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52 #2 0x0000000805913ca9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65 #3 0x0000000805639554 in _thread_exit ( fname=0x80563ac36 "/usr/src/lib/libthr/thread/thr_mutex.c", lineno=155, msg=0x7fffffffd1c0 "mutex 0x800632000 own 0x1885c 0x1885c is on list 0x0 0x80d46ebc0") at /usr/src/lib/libthr/thread/thr_exit.c:182 #4 0x000000080562fe36 in mutex_assert_not_owned (m=0x800632000) at /usr/src/lib/libthr/thread/thr_mutex.c:155 #5 0x0000000805630009 in enqueue_mutex (curthread=0x80cc15000, m=0x800632000) at /usr/src/lib/libthr/thread/thr_mutex.c:400 #6 0x0000000805631665 in mutex_lock_sleep (curthread=0x80cc15000, m=0x800632000, abstime=0x7fffffffd358) at /usr/src/lib/libthr/thread/thr_mutex.c:535 #7 0x0000000805630280 in mutex_lock_common (m=0x800632000, abstime=0x7fffffffd358, cvattach=0) at /usr/src/lib/libthr/thread/thr_mutex.c:553 #8 0x000000080562f6be in __pthread_mutex_timedlock (mutex=0x810a00008, abstime=0x7fffffffd358) at /usr/src/lib/libthr/thread/thr_mutex.c:583 ... gdb) f 6 #6 0x0000000805631665 in mutex_lock_sleep (curthread=0x80cc15000, m=0x800632000, abstime=0x7fffffffd358) at /usr/src/lib/libthr/thread/thr_mutex.c:535 535 enqueue_mutex(curthread, m); (gdb) p *curthread $1 = {tid = 100444, lock = {m_owner = 0, m_flags = 0, m_ceilings = {0, 0}, m_spare = {0, 0, 0, 0}}, cycle = 0, locklevel = 0, critical_count = 0, sigblock = 0, tle = { tqe_next = 0x0, tqe_prev = 0x805841000 <_thread_list>}, gcle = {tqe_next = 0x0, tqe_prev = 0x0}, hle = {le_next = 0x0, le_prev = 0x80584b3c0}, wle = {tqe_next = 0x0, tqe_prev = 0x0}, refcount = 0, start_routine = 0x0, arg = 0x0, attr = {sched_policy = 2, sched_inherit = 4, prio = 0, suspend = 0, flags = 258, stackaddr_attr = 0x7ffffdfff000, stacksize_attr = 33554432, guardsize_attr = 4096, cpuset = 0x0, cpusetsize = 0}, cancel_enable = 1, cancel_pending = 0, cancel_point = 0, no_cancel = 0, cancel_async = 0, cancelling = 0, sigmask = {__bits = {0, 0, 0, 0}}, unblock_sigcancel = 0, in_sigsuspend = 0, deferred_siginfo = {si_signo = 0, si_errno = 0, si_code = 0, si_pid = 0, si_uid = 0, si_status = 0, si_addr = 0x0, si_value = {sival_int = 0, sival_ptr = 0x0, sigval_int = 0, sigval_ptr = 0x0}, _reason = {_fault = {_trapno = 0}, _timer = { _timerid = 0, _overrun = 0}, _mesgq = {_mqd = 0}, _poll = {_band = 0}, __spare__ = { __spare1__ = 0, __spare2__ = {0, 0, 0, 0, 0, 0, 0}}}}, deferred_sigmask = {__bits = { 0, 0, 0, 0}}, deferred_sigact = {__sigaction_u = {__sa_handler = 0x0, __sa_sigaction = 0x0}, sa_flags = 0, sa_mask = {__bits = {0, 0, 0, 0}}}, deferred_run = 0, force_exit = 0, state = PS_RUNNING, error = 0, joiner = 0x0, flags = 0, tlflags = 2, mq = {{tqh_first = 0x0, tqh_last = 0x80cc151a0}, {tqh_first = 0x0, tqh_last = 0x80cc151b0}, {tqh_first = 0x0, tqh_last = 0x80cc151c0}, {tqh_first = 0x0, tqh_last = 0x80cc151d0}}, ret = 0x0, specific = 0x800631000, specific_data_count = 5, rdlock_count = 0, rtld_bits = 0, tcb = 0x8006df108, cleanup = 0x0, ex = { exception_class = 0, exception_cleanup = 0x0, private_1 = 0, private_2 = 0}, unwind_stackend = 0x7ffffffff000, unwind_disabled = 0, magic = 3499860245, report_events = 0, event_mask = 0, event_buf = {event = TD_EVENT_NONE, th_p = 0, data = 0}, wchan = 0x0, mutex_obj = 0x0, will_sleep = 0, nwaiter_defer = 0, defer_waiters = { 0x0 }, wake_addr = 0x80584b0c8, sleepqueue = 0x80cc14040} (gdb) f 8 #8 0x000000080562f6be in __pthread_mutex_timedlock (mutex=0x810a00008, abstime=0x7fffffffd358) at /usr/src/lib/libthr/thread/thr_mutex.c:583 583 ret = mutex_lock_common(m, abstime, 0); (gdb) p *mutex $2 = (pthread_mutex_t) 0x8000000000000001 (gdb) p m $3 = (struct pthread_mutex *) 0x800632000 (gdb) p *m $4 = {m_lock = {m_owner = 100444, m_flags = 1, m_ceilings = {0, 0}, m_spare = {0, 0, 0, 0}}, m_flags = 1, m_owner = 100444, m_count = 0, m_spinloops = 0, m_yieldloops = 0, m_qe = { tqe_next = 0x0, tqe_prev = 0x0}, m_pqe = {tqe_next = 0x0, tqe_prev = 0x0}} Thank you