From owner-freebsd-threads@FreeBSD.ORG Tue Oct 28 16:19:28 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C6E791065672 for ; Tue, 28 Oct 2008 16:19:28 +0000 (UTC) (envelope-from sbartnikowski@barkinglizards.com) Received: from pluto.phpwebhosting.com (pluto.phpwebhosting.com [69.0.209.128]) by mx1.freebsd.org (Postfix) with SMTP id 7EBA08FC1A for ; Tue, 28 Oct 2008 16:19:28 +0000 (UTC) (envelope-from sbartnikowski@barkinglizards.com) Received: (qmail 30687 invoked from network); 28 Oct 2008 15:52:42 -0000 Received: from unknown (HELO OCTOBER) (216.138.73.11) by pluto.phpwebhosting.com with SMTP; Tue, 28 Oct 2008 11:52:42 -0400 From: "Stephen Bartnikowski" To: Date: Tue, 28 Oct 2008 10:52:45 -0500 Organization: Barking Lizards Technologies Message-ID: <5B0A3FB3D06E41849325AD3EA98CAD55@OCTOBER> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 11 Thread-Index: Ack5FTfqj+Y3z0v1SliCeTc+JtC5+w== X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 Subject: deadlock in fork call? X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: sbartnikowski@barkinglizards.com List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Oct 2008 16:19:29 -0000 Hi guys, To start, I'm not reporting a bug here. I'm trying to get an idea if I'm on the right track. Also, apologies on the lengthy email. Oh and this likely relevant. I'm running FreeBSD 6.3 RELEASE that I installed on July 21. I don't believe any updates have been performed since. I've got a deadlock situation between a fork call on the main thread and another thread. I already know I'm venturing into shady territory when I fork a multithreaded process. However, I don't have a good understanding of exactly what's going on there, which is where I'm hoping you guys can help. Stack dumps are at the bottom of this email. Idea 1: Thread #5 blows up and in its bad state is stuck on a mutex for whatever reason. Thread #1 in the fork call is attempting to reinitialize each mutex in the process space and deadlocks because Thread #5 is hosed and has one of those mutexes. Idea 2: Thread #5 blows up at the same time that Thread #1 makes the fork call. Since at this point in FreeBSD (6.3), the Giant Lock logic is still in there, and these two threads deadlock each other on pthread_mutexattr_init calls. Note that my forked process logic is followed by an execv call, although I just noticed that if execv fails for whatever reason, I'm likely doing some unsafe logging calls before that forked process quits. Any input on whether Idea 1 or Idea 2 are valid? Thank you! - Stephen Here's the main thread (#1) doing the fork call. It's blocked on that pthread_mutexattr_init call. #0 0x28109198 in pthread_sigmask () from /lib/libpthread.so.2 #1 0x28109148 in sigprocmask () from /lib/libpthread.so.2 #2 0x2811360c in pthread_mutexattr_init () from /lib/libpthread.so.2 #3 0x281062db in fork () from /lib/libpthread.so.2 #4 0x2850e04c in blt::Fork () at wrapunix.cpp:978 #5 0x08073e7b in cerebro::ModuleManager::loadModule (this=0x810af40, rModule=@0x816a200) at ModuleManager.cpp:500 #6 0x08073f54 in cerebro::ModuleManager::loadModules (this=0x810af40, mqipctok=@0x80ef134, piddir=@0x80ef124) at ModuleManager.cpp:536 #7 0x08064e26 in cerebro::Cerebro::main (this=0x80ef100, argc=4, argv=0xbfbfe9d4) at Cerebro.cpp:721 #8 0x08069e6a in main (argc=4, argv=0xbfbfe9d4) at Cerebro.cpp:1934 And here's the second (corrupted) thread (#5), that I'm 90% sure is caused by a third-party logging library, unfortunately. #0 0x28114fbf in pthread_mutexattr_init () from /lib/libpthread.so.2 #1 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 #2 0x28112cc6 in pthread_mutex_lock () from /lib/libpthread.so.2 #3 0x2811d3f5 in __error () from /lib/libpthread.so.2 #4 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 #5 0x28112cc6 in pthread_mutex_lock () from /lib/libpthread.so.2 #6 0x2810831a in _spinunlock () from /lib/libpthread.so.2 #7 0x289c5d22 in _UTF8_init () from /lib/libc.so.6 #8 0x28a42580 in _thread_autoinit_dummy_decl_stub () from /lib/libc.so.6 #9 0x0813c280 in ?? () #10 0xbf8fda38 in ?? () #11 0x289c5cc9 in _UTF8_init () from /lib/libc.so.6 #12 0x28a2f593 in sys_nsig () from /lib/libc.so.6 #13 0x2894e5c4 in ?? () from /usr/lib/libstdc++.so.5 #14 0xbf8fd9c8 in ?? () #15 0x2892db3d in operator delete () from /usr/lib/libstdc++.so.5 #16 0x2892db51 in operator delete () from /usr/lib/libstdc++.so.5 #17 0x288ed6fb in std::string::_Rep::_M_destroy () from /usr/lib/libstdc++.so.5 #18 0x28110e29 in _pthread_mutex_trylock () from /lib/libpthread.so.2 #19 0x2811d3f5 in __error () from /lib/libpthread.so.2 #20 0x280fa170 in ?? () #21 0x00000000 in ?? () #22 0xbf8fdb64 in ?? () #23 0x280d46fc in symlook_obj () from /libexec/ld-elf.so.1