From owner-freebsd-threads@FreeBSD.ORG Mon Sep 29 11:06:59 2008 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 30010106568C for ; Mon, 29 Sep 2008 11:06:59 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1EDCB8FC13 for ; Mon, 29 Sep 2008 11:06:59 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m8TB6xd0040966 for ; Mon, 29 Sep 2008 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m8TB6wrf040962 for freebsd-threads@FreeBSD.org; Mon, 29 Sep 2008 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 29 Sep 2008 11:06:58 GMT Message-Id: <200809291106.m8TB6wrf040962@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Sep 2008 11:06:59 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o threa/127225 threads bug in lib/libthr/thread/thr_init.c o threa/126950 threads [patch] rtld(1): rtld malloc is thread-unsafe o kern/126128 threads [patch] pthread_condattr_getpshared is broken o threa/122923 threads 'nice' does not prevent background process from steali o threa/121336 threads lang/neko threading ok on UP, broken on SMP (FreeBSD 7 o threa/118715 threads kse problem o threa/116668 threads can no longer use jdk15 with libthr on -stable SMP o threa/116181 threads /dev/io-related io access permissions are not propagat o threa/115211 threads pthread_atfork misbehaves in initial thread o threa/110636 threads [request] gdb(1): using gdb with multi thread applicat o threa/110306 threads apache 2.0 segmentation violation when calling gethost o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/101323 threads fork(2) in threaded programs broken. s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/83914 threads [libc] popen() doesn't work in static threaded program o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/80435 threads panic on high loads o threa/79887 threads [patch] freopen() isn't thread-safe o threa/79683 threads svctcp_create() fails if multiple threads call at the s threa/76694 threads fork cause hang in dup()/close() function in child (-l s threa/76690 threads fork hang in child for -lc_r o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/70975 threads [sysvipc] unexpected and unreliable behaviour when usi s threa/69020 threads pthreads library leaks _gc_mutex s threa/49087 threads Signals lost in programs linked with libc_r s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/34536 threads accept() blocks other threads s kern/32295 threads [libc_r] [patch] pthread(3) dont dequeue signals s threa/30464 threads pthread mutex attributes -- pshared s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o 38 problems total. From owner-freebsd-threads@FreeBSD.ORG Mon Sep 29 16:59:48 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 744C910656A2 for ; Mon, 29 Sep 2008 16:59:48 +0000 (UTC) (envelope-from dchhetri@panasas.com) Received: from laguna.int.panasas.com (gw-ca.panasas.com [66.104.249.162]) by mx1.freebsd.org (Postfix) with ESMTP id 58C648FC0C for ; Mon, 29 Sep 2008 16:59:47 +0000 (UTC) (envelope-from dchhetri@panasas.com) Received: from [172.17.132.94] ([172.17.132.94]) by laguna.int.panasas.com with Microsoft SMTPSVC(6.0.3790.3959); Mon, 29 Sep 2008 09:58:32 -0700 Message-ID: <48E10978.2090907@panasas.com> Date: Mon, 29 Sep 2008 09:59:36 -0700 From: Dilip Chhetri User-Agent: Mozilla Thunderbird 1.0.2-6 (X11/20050513) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Tijl Coosemans References: <48DD32D2.2060304@panasas.com> <200809262331.29353.tijl@ulyssis.org> In-Reply-To: <200809262331.29353.tijl@ulyssis.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 29 Sep 2008 16:58:32.0615 (UTC) FILETIME=[9AC7C370:01C92254] Cc: freebsd-threads@freebsd.org Subject: Re: getting stack trace for other thread on the same process : libthr X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Sep 2008 16:59:48 -0000 Tijl Coosemans wrote: > On Friday 26 September 2008 21:06:58 Dilip Chhetri wrote: > >>Question >>-------- >> My program is linked with libthr in FreeBSD-7.0. The program has >>in the order of 20 threads, and a designated monitoring thread at >>some point wants to know what are other/stuck threads doing. This >>needs to be done by printing stack backtrace for the thread to >>stdout. >> >> I understand pthread_t structure has pointer to the target >>thread's stack, but to get the trace I need to know value of >>stack-pointer register and base-pointer register. I looked at the >>code and I don't find any mechanism by which I could read the target >>threads register context (because it all resides within kernel thread >>structure). Further code study reveals that kernel_thread->td_frame >>contains the register context for a thread, but is valid only when >>the thread is executing/sleeping inside the kernel. >> >> Is there anything I'm missing here ? Is there an easy way to >>traverse stack for some thread with in the same process. >> >> I considered/considering following approaches, >>a) use PTRACE >> ruled out, because you can't trace the process from within the >> same process >> >>b) somehow temporarily stop the target-thread and read td_frame by >> traversing kernel data structure through /dev/kmem. After doing >> stack traversal resume the target thread. >> >> >>Detailed problem background >>-------------------------- >> We have this process X with ~20 threads, each processing some >>requests. One of them is designated as monitoring/dispatcher thread. >>When a new request arrives, dispatcher thread tries to queue the task >>to idle thread. But if all threads are busy processing requests, the >>dispatcher thread is supposed to print the stack back trace for each >>of the busy thread. This is our *debugging* mechanism to find >>potential fault-points. >> >> In FreeBSD-4.6.2, we hacked libc_r:pthread_t to achieve our goal. >>But in FreeBSD-7.0, we decided to use libthr and hack doesn't seem to >>be easy. >> >>Target setup >>------------ >> * SMP : around 8 CPU >> * process : it's going to be run as root and have around ~20 threads > > > You could try registering a signal handler for SIGUSR1 that prints a > stack backtrace using the stack pointer in the sigcontext and then call > pthread_kill(SIGUSR1) on whichever thread you want a backtrace of. Thanks, but as I mentioned it's a network based program and it may be sleeping/stuck in syscall for some packets, in this case pthread_kill will not work because signals are delivered only when you return from syscall (that's what I haved learned from old UNIX books in my college). From owner-freebsd-threads@FreeBSD.ORG Mon Sep 29 20:09:57 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F0031065690 for ; Mon, 29 Sep 2008 20:09:57 +0000 (UTC) (envelope-from tijl@ulyssis.org) Received: from mailrelay011.isp.belgacom.be (mailrelay011.isp.belgacom.be [195.238.6.178]) by mx1.freebsd.org (Postfix) with ESMTP id 2CA088FC25 for ; Mon, 29 Sep 2008 20:09:56 +0000 (UTC) (envelope-from tijl@ulyssis.org) X-Belgacom-Dynamic: yes X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AuYEADLT4EhR9S8L/2dsb2JhbACBYrowgWc Received: from 11.47-245-81.adsl-dyn.isp.belgacom.be (HELO kalimero.kotnet.org) ([81.245.47.11]) by relay.skynet.be with ESMTP; 29 Sep 2008 22:09:55 +0200 Received: from kalimero.kotnet.org (kalimero.kotnet.org [127.0.0.1]) by kalimero.kotnet.org (8.14.3/8.14.3) with ESMTP id m8TK8UnN023377; Mon, 29 Sep 2008 22:08:30 +0200 (CEST) (envelope-from tijl@ulyssis.org) From: Tijl Coosemans To: Dilip Chhetri Date: Mon, 29 Sep 2008 22:08:27 +0200 User-Agent: KMail/1.9.10 References: <48DD32D2.2060304@panasas.com> <200809262331.29353.tijl@ulyssis.org> <48E10978.2090907@panasas.com> In-Reply-To: <48E10978.2090907@panasas.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200809292208.29315.tijl@ulyssis.org> Cc: freebsd-threads@freebsd.org Subject: Re: getting stack trace for other thread on the same process : libthr X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Sep 2008 20:09:57 -0000 On Monday 29 September 2008 18:59:36 Dilip Chhetri wrote: > Tijl Coosemans wrote: >> On Friday 26 September 2008 21:06:58 Dilip Chhetri wrote: >> >>> Question >>> -------- >>> My program is linked with libthr in FreeBSD-7.0. The program has >>> in the order of 20 threads, and a designated monitoring thread at >>> some point wants to know what are other/stuck threads doing. This >>> needs to be done by printing stack backtrace for the thread to >>> stdout. >>> >>> I understand pthread_t structure has pointer to the target >>> thread's stack, but to get the trace I need to know value of >>> stack-pointer register and base-pointer register. I looked at the >>> code and I don't find any mechanism by which I could read the >>> target threads register context (because it all resides within >>> kernel thread structure). Further code study reveals that >>> kernel_thread->td_frame contains the register context for a thread, >>> but is valid only when the thread is executing/sleeping inside the >>> kernel. >>> >>> Is there anything I'm missing here ? Is there an easy way to >>> traverse stack for some thread with in the same process. >>> >>> I considered/considering following approaches, >>> a) use PTRACE >>> ruled out, because you can't trace the process from within the >>> same process >>> >>> b) somehow temporarily stop the target-thread and read td_frame by >>> traversing kernel data structure through /dev/kmem. After doing >>> stack traversal resume the target thread. >>> >>> >>> Detailed problem background >>> -------------------------- >>> We have this process X with ~20 threads, each processing some >>> requests. One of them is designated as monitoring/dispatcher >>> thread. When a new request arrives, dispatcher thread tries to >>> queue the task to idle thread. But if all threads are busy >>> processing requests, the dispatcher thread is supposed to print the >>> stack back trace for each of the busy thread. This is our >>> *debugging* mechanism to find potential fault-points. >>> >>> In FreeBSD-4.6.2, we hacked libc_r:pthread_t to achieve our goal. >>> But in FreeBSD-7.0, we decided to use libthr and hack doesn't seem >>> to be easy. >>> >>> Target setup >>> ------------ >>> * SMP : around 8 CPU >>> * process : it's going to be run as root and have around ~20 >>> threads >> >> You could try registering a signal handler for SIGUSR1 that prints a >> stack backtrace using the stack pointer in the sigcontext and then >> call pthread_kill(SIGUSR1) on whichever thread you want a backtrace >> of. > > Thanks, but as I mentioned it's a network based program and it may be > sleeping/stuck in syscall for some packets, in this case pthread_kill > will not work because signals are delivered only when you return from > syscall (that's what I haved learned from old UNIX books in my > college). Those kind of syscalls are usually interruptable though. Depending on the SA_RESTART flag they are then either aborted and return EINTR or restarted (or return partial success). See the sigaction(2) manpage. From owner-freebsd-threads@FreeBSD.ORG Mon Sep 29 21:43:25 2008 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3FA701065691 for ; Mon, 29 Sep 2008 21:43:25 +0000 (UTC) (envelope-from dchhetri@panasas.com) Received: from laguna.int.panasas.com (gw-ca.panasas.com [66.104.249.162]) by mx1.freebsd.org (Postfix) with ESMTP id A53008FC15 for ; Mon, 29 Sep 2008 21:43:24 +0000 (UTC) (envelope-from dchhetri@panasas.com) Received: from [172.17.132.94] ([172.17.132.94]) by laguna.int.panasas.com with Microsoft SMTPSVC(6.0.3790.3959); Mon, 29 Sep 2008 14:42:07 -0700 Message-ID: <48E14BF0.1050108@panasas.com> Date: Mon, 29 Sep 2008 14:43:12 -0700 From: Dilip Chhetri User-Agent: Mozilla Thunderbird 1.0.2-6 (X11/20050513) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Tijl Coosemans References: <48DD32D2.2060304@panasas.com> <200809262331.29353.tijl@ulyssis.org> <48E10978.2090907@panasas.com> <200809292208.29315.tijl@ulyssis.org> In-Reply-To: <200809292208.29315.tijl@ulyssis.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 29 Sep 2008 21:42:07.0947 (UTC) FILETIME=[38B561B0:01C9227C] Cc: freebsd-threads@freebsd.org Subject: Re: getting stack trace for other thread on the same process : libthr X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Sep 2008 21:43:25 -0000 Tijl Coosemans wrote: > On Monday 29 September 2008 18:59:36 Dilip Chhetri wrote: > >>Tijl Coosemans wrote: >> >>>On Friday 26 September 2008 21:06:58 Dilip Chhetri wrote: >>> >>> >>>>Question >>>>-------- >>>> My program is linked with libthr in FreeBSD-7.0. The program has >>>>in the order of 20 threads, and a designated monitoring thread at >>>>some point wants to know what are other/stuck threads doing. This >>>>needs to be done by printing stack backtrace for the thread to >>>>stdout. >>>> >>>> I understand pthread_t structure has pointer to the target >>>>thread's stack, but to get the trace I need to know value of >>>>stack-pointer register and base-pointer register. I looked at the >>>>code and I don't find any mechanism by which I could read the >>>>target threads register context (because it all resides within >>>>kernel thread structure). Further code study reveals that >>>>kernel_thread->td_frame contains the register context for a thread, >>>>but is valid only when the thread is executing/sleeping inside the >>>>kernel. >>>> >>>> Is there anything I'm missing here ? Is there an easy way to >>>>traverse stack for some thread with in the same process. >>>> >>>> I considered/considering following approaches, >>>>a) use PTRACE >>>> ruled out, because you can't trace the process from within the >>>> same process >>>> >>>>b) somehow temporarily stop the target-thread and read td_frame by >>>> traversing kernel data structure through /dev/kmem. After doing >>>> stack traversal resume the target thread. >>>> >>>> >>>>Detailed problem background >>>>-------------------------- >>>> We have this process X with ~20 threads, each processing some >>>>requests. One of them is designated as monitoring/dispatcher >>>>thread. When a new request arrives, dispatcher thread tries to >>>>queue the task to idle thread. But if all threads are busy >>>>processing requests, the dispatcher thread is supposed to print the >>>>stack back trace for each of the busy thread. This is our >>>>*debugging* mechanism to find potential fault-points. >>>> >>>> In FreeBSD-4.6.2, we hacked libc_r:pthread_t to achieve our goal. >>>>But in FreeBSD-7.0, we decided to use libthr and hack doesn't seem >>>>to be easy. >>>> >>>>Target setup >>>>------------ >>>> * SMP : around 8 CPU >>>> * process : it's going to be run as root and have around ~20 >>>> threads >>> >>>You could try registering a signal handler for SIGUSR1 that prints a >>>stack backtrace using the stack pointer in the sigcontext and then >>>call pthread_kill(SIGUSR1) on whichever thread you want a backtrace >>>of. >> >>Thanks, but as I mentioned it's a network based program and it may be >>sleeping/stuck in syscall for some packets, in this case pthread_kill >>will not work because signals are delivered only when you return from >>syscall (that's what I haved learned from old UNIX books in my >>college). > > > Those kind of syscalls are usually interruptable though. Depending on > the SA_RESTART flag they are then either aborted and return EINTR or > restarted (or return partial success). See the sigaction(2) manpage. thanks. I will give that a try, maybe it will work 90% of the time for us. Thats much better than having nothing or something that is too complicated to implement. Thanks once again.