From owner-freebsd-threads@FreeBSD.ORG Sun Jul 4 20:58:29 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 886BD16A4CE for ; Sun, 4 Jul 2004 20:58:29 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3556343D1D for ; Sun, 4 Jul 2004 20:58:27 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i64Kw5qM011265 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 4 Jul 2004 16:58:05 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i64KvwQv014978; Sun, 4 Jul 2004 16:57:58 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16616.28502.692389.458282@grasshopper.cs.duke.edu> Date: Sun, 4 Jul 2004 16:57:58 -0400 (EDT) To: Julian Elischer In-Reply-To: References: <16613.53106.413179.808734@grasshopper.cs.duke.edu> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: freebsd-threads@freebsd.org cc: Andrew Gallatin Subject: Re: odd KSE panic X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jul 2004 20:58:29 -0000 Julian Elischer writes: > When one thread calls exit() it marks the fact that the process is > exiting, and then tries to wakeup all the other threads, and then > suspends itself. The other threads, when awoken are supposed to notice > what's going on and abort whatever they are doing and when they release > all their resources, (by unrolling back to the user boundary) they are > supposed to call thread_exit(). The last one out is supposed to > wakeyup the original thread that called exit(), which can then proceed > on the basis that it is now the only remaining thread. > > If there are threads waiting in uninterruptble sleeps then the process > as a whole can not exit until they have finished sleeping and come back > to the user boundary and called thread_exit(). > > None of the three threads you show is in exit, or even anything related > to exit. > Thank you for this explaination.. The way that threads exit is one of the things I've never understood about KSE. > > ummmm nope.. where is mx_free? > Unreleased 3rd party device driver.. Drew From owner-freebsd-threads@FreeBSD.ORG Sun Jul 4 21:20:11 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 519D016A4CE for ; Sun, 4 Jul 2004 21:20:11 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 10D9343D1D for ; Sun, 4 Jul 2004 21:20:11 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i64LKAqM013281 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 4 Jul 2004 17:20:10 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i64LK5lV015002; Sun, 4 Jul 2004 17:20:05 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16616.29829.132376.76817@grasshopper.cs.duke.edu> Date: Sun, 4 Jul 2004 17:20:05 -0400 (EDT) To: freebsd-threads@freebsd.org In-Reply-To: <16613.45444.528419.643022@grasshopper.cs.duke.edu> References: <16613.45444.528419.643022@grasshopper.cs.duke.edu> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Subject: Re: odd KSE panic X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jul 2004 21:20:11 -0000 Andrew Gallatin writes: > > The problem happens when exiting. When main application thread > decides to exit, it does an ioctl into the driver to wakeup the > sleeping worker thread. The worker thread thread wakes up, and then > exits, then the main thread closes his file descriptor and exits. > > The problem I'm seeing is that I get a panic like the following when > using KSE. (A linux binary works fine, ioctls are translated..) > The problem turned out to be that the worker thread was sleeping in cv_wait_sig() on a cv which was used elsewhere in the driver. When I fixed this, pretty much everything got better. I still don't understand exactly what happened. I have no idea if the worker woke too early, or if the other place this cv was (mis)used was where the early wake happened. (this would be where mx_free() is called). Anyway, it no longer crashes the machine. Sorry to have wasted your time. Now to figure out why libthr does pthread_cond_signal() in my scenario 47us faster than libpthread... (ULE, 1 HTT P4 running SMP kernel) Scenario is that the mainline thread sleeps waiting for a packet (using pthread_cond_timedwait()) and the worker thread is asleep in kernel. When a packet arrives, the worker wakes up, returns from the ioctl, does a pthread_cond_signal() to wakeup the mainline thread, and goes back into the kernel to sleep (via an ioctl). This is the sort of scenario where I thought KSE would be faster than a 1:1 lib.. Drew From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 07:31:55 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 893D916A4CE for ; Mon, 5 Jul 2004 07:31:55 +0000 (GMT) Received: from sccrmhc11.comcast.net (sccrmhc11.comcast.net [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3D8F343D3F for ; Mon, 5 Jul 2004 07:31:55 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc11) with ESMTP id <20040705073154011009natve>; Mon, 5 Jul 2004 07:31:54 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id AAA42352; Mon, 5 Jul 2004 00:31:51 -0700 (PDT) Date: Mon, 5 Jul 2004 00:31:50 -0700 (PDT) From: Julian Elischer To: Andrew Gallatin In-Reply-To: <16616.29829.132376.76817@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 07:31:55 -0000 On Sun, 4 Jul 2004, Andrew Gallatin wrote: > > Andrew Gallatin writes: > > > > The problem happens when exiting. When main application thread > > decides to exit, it does an ioctl into the driver to wakeup the > > sleeping worker thread. The worker thread thread wakes up, and then > > exits, then the main thread closes his file descriptor and exits. > > > > The problem I'm seeing is that I get a panic like the following when > > using KSE. (A linux binary works fine, ioctls are translated..) > > > > The problem turned out to be that the worker thread was sleeping in > cv_wait_sig() on a cv which was used elsewhere in the driver. When I > fixed this, pretty much everything got better. I still don't > understand exactly what happened. I have no idea if the worker woke > too early, or if the other place this cv was (mis)used was where the > early wake happened. (this would be where mx_free() is called). > > Anyway, it no longer crashes the machine. Sorry to have wasted your > time. > > Now to figure out why libthr does pthread_cond_signal() in my scenario > 47us faster than libpthread... (ULE, 1 HTT P4 running SMP kernel) > > Scenario is that the mainline thread sleeps waiting for a packet > (using pthread_cond_timedwait()) and the worker thread is asleep in > kernel. When a packet arrives, the worker wakes up, returns from the > ioctl, does a pthread_cond_signal() to wakeup the mainline thread, and > goes back into the kernel to sleep (via an ioctl). This is the sort > of scenario where I thought KSE would be faster than a 1:1 lib.. Actually it is the sort of scenario where it is not clear what is best ;-) The mainline thread is asleep in userland. thus to become active, if there is no running kernel thread to run it, one needs to be woken up from the kernel and made to upcall to userspace to be assigned to run the thread. This is pretty much what is happenning in M:M threads. (1:1 is in my opinion a non threadded app :-) Except that in KSE there is more overhead to do it.. One thing that could be tuned in KSE to make it a lot faster is to reverse the order that the threads are run in the following way: ------ currently: Kernel thread 1 returns from waiting for the packet, and as user thread A, signals (enters) the UTS to make user thread B runnable. The UTS realises that there is no kernel thread currently running to run thread B, and enters the kernel to wake one up, and returns to userland, in order to re-enter thread A and go back into teh kernel to wait for the next packet. Kernel thread 2 is started up some time after it is awoken and enters userland via an upcall and is assigned thread B to run. This is all assuming there are 2 (or more CPUS). If there is only one CPU then when user thread A sleeps in the kernel, the kernel thread (1) is allowed to upcall back to the UTS and take on user thread B. ------ the changed version: Kernel thread 1 returns from waiting for the packet, and as user thread A, signals (enters) the UTS to make user thread B runnable. The UTS realises that there is no kernel thread currently running to run thread B it enters the kernel briefly to make another thread wake up and upcall, and then suspends thread A, and takes on thread B. Eventually th e upcall occurs and thread A is assigned to the newly awoken thread and re-enters the kernel, waitign for the next kernel. ------------ Dan, can the 2nd version happen now? can Drew make it happen by assigning a higher priority to thread B? > > Drew > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 11:01:57 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D66B816A4D1 for ; Mon, 5 Jul 2004 11:01:57 +0000 (GMT) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id B6C5743D2D for ; Mon, 5 Jul 2004 11:01:57 +0000 (GMT) (envelope-from owner-bugmaster@freebsd.org) Received: from freefall.freebsd.org (peter@localhost [127.0.0.1]) by freefall.freebsd.org (8.12.11/8.12.11) with ESMTP id i65B1vIT055647 for ; Mon, 5 Jul 2004 11:01:57 GMT (envelope-from owner-bugmaster@freebsd.org) Received: (from peter@localhost) by freefall.freebsd.org (8.12.11/8.12.11/Submit) id i65B1vQ0055641 for freebsd-threads@freebsd.org; Mon, 5 Jul 2004 11:01:57 GMT (envelope-from owner-bugmaster@freebsd.org) Date: Mon, 5 Jul 2004 11:01:57 GMT Message-Id: <200407051101.i65B1vQ0055641@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: peter set sender to owner-bugmaster@freebsd.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 11:01:58 -0000 Current FreeBSD problem reports Critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/06/13] kern/19247 threads uthread_sigaction.c does not do anything s [2004/03/15] kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/un o [2004/04/22] threads/65883threads libkse's sigwait does not work after fork 3 problems total. Serious problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/07/18] kern/20016 threads pthreads: Cannot set scheduling timer/Can o [2000/08/26] misc/20861 threads libc_r does not honor socket timeouts o [2001/01/20] bin/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVT o [2001/01/25] bin/24632 threads libc_r delicate deviation from libc in ha o [2001/01/25] misc/24641 threads pthread_rwlock_rdlock can deadlock o [2001/11/26] bin/32295 threads pthread dont dequeue signals o [2002/02/01] i386/34536 threads accept() blocks other threads o [2002/05/25] kern/38549 threads the procces compiled whith pthread stoppe o [2002/06/27] bin/39922 threads [PATCH?] Threaded applications executed w o [2002/08/04] misc/41331 threads Pthread library open sets O_NONBLOCK flag o [2003/03/02] bin/48856 threads Setting SIGCHLD to SIG_IGN still leaves z o [2003/03/10] bin/49087 threads Signals lost in programs linked with libc o [2003/05/08] bin/51949 threads thread in accept cannot be cancelled 13 problems total. Non-critical problems S Submitted Tracker Resp. Description ------------------------------------------------------------------------------- o [2000/05/26] misc/18824 threads gethostbyname is not thread safe o [2000/10/21] misc/22190 threads A threaded read(2) from a socketpair(2) f o [2001/09/09] bin/30464 threads pthread mutex attributes -- pshared o [2002/05/02] bin/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwri s [2002/07/16] threads/40671threads pthread_cancel doesn't remove thread from 5 problems total. From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 11:44:26 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2CF4416A4CE for ; Mon, 5 Jul 2004 11:44:26 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id A23C143D39 for ; Mon, 5 Jul 2004 11:44:25 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i65BiBfP012546; Mon, 5 Jul 2004 07:44:11 -0400 (EDT) Date: Mon, 5 Jul 2004 07:44:11 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 11:44:26 -0000 On Mon, 5 Jul 2004, Julian Elischer wrote: > > On Sun, 4 Jul 2004, Andrew Gallatin wrote: > > > The problem turned out to be that the worker thread was sleeping in > > cv_wait_sig() on a cv which was used elsewhere in the driver. When I > > fixed this, pretty much everything got better. I still don't > > understand exactly what happened. I have no idea if the worker woke > > too early, or if the other place this cv was (mis)used was where the > > early wake happened. (this would be where mx_free() is called). > > > > Anyway, it no longer crashes the machine. Sorry to have wasted your > > time. > > > > Now to figure out why libthr does pthread_cond_signal() in my scenario > > 47us faster than libpthread... (ULE, 1 HTT P4 running SMP kernel) > > > > Scenario is that the mainline thread sleeps waiting for a packet > > (using pthread_cond_timedwait()) and the worker thread is asleep in > > kernel. When a packet arrives, the worker wakes up, returns from the > > ioctl, does a pthread_cond_signal() to wakeup the mainline thread, and > > goes back into the kernel to sleep (via an ioctl). This is the sort > > of scenario where I thought KSE would be faster than a 1:1 lib.. ULE & HTT & KSE is probably the most inefficient combination... I find true SMP & 4BSD to be better for KSE (perhaps after julian finishes his cleanup, things will be better). > Actually it is the sort of scenario where it is not clear what is best > ;-) > > The mainline thread is asleep in userland. thus to become active, if > there is no running kernel thread to run it, one needs to be woken up > from the kernel and made to upcall to userspace to be assigned to run > the thread. This is pretty much what is happenning in M:M threads. (1:1 > is in my opinion a non threadded app :-) Except that in KSE there is > more overhead to do it.. One thing that could be tuned in KSE to make it > a lot faster is to reverse the order that the threads are run in the > following way: > > ------ currently: > > Kernel thread 1 returns from waiting for the packet, and as user thread > A, signals (enters) the UTS to make user thread B runnable. The UTS > realises that there is no kernel thread currently running to run thread > B, and enters the kernel to wake one up, and returns to userland, in > order to re-enter thread A and go back into teh kernel to wait for the > next packet. Actually, the UTS doesn't really think of it in terms of creating new kernel threads. It only sees idle and non-idle KSEs. > Kernel thread 2 is started up some time after it is awoken and enters > userland via an upcall and is assigned thread B to run. > > This is all assuming there are 2 (or more CPUS). > If there is only one CPU then when user thread A sleeps in the kernel, > the kernel thread (1) is allowed to upcall back to the UTS and take on > user thread B. This is true regardless of the number of CPUs, unless you have system scope threads. Any blocking in the kernel will result in an upcall. > ------ the changed version: > > Kernel thread 1 returns from waiting for the packet, and as user thread > A, signals (enters) the UTS to make user thread B runnable. The UTS > realises that there is no kernel thread currently running to run thread > B it enters the kernel briefly to make another thread wake up and > upcall, and then suspends thread A, and takes on thread B. > Eventually th e upcall occurs and thread A is assigned to the newly > awoken thread and re-enters the kernel, waitign for the next kernel. > > ------------ > > Dan, can the 2nd version happen now? > can Drew make it happen by assigning a higher priority to thread B? No, not yet; pthread_cond_signal() and pthread_mutex_unlock() are not currently preemption points. I'm not sure that you would want pthread_cond_signal() to be a preemption point -- I'd imagine that a lot of applications would do something like this: /* sleeping */ pthread_mutex_lock(&m); foo->sleeping_count++; pthread_cond_wait(&cv, &m); pthread_mutex_unlock(&m); /* waking up */ pthread_mutex_lock(&m); if (foo->sleeping_count > 0) { foo->sleeping_count--; pthread_cond_signal(&cv); } pthread_mutex_unlock(&m); If you make pthread_cond_signal() a preemption point, then you would get some ping-pong-like effect. The newly awoken thread would block again on the mutex. With a small number of threads, it probably doesn't make sense to have more than one KSE (unless they are mostly CPU-bound). In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads. Each time a thread sleeps, the KSE enters the kernel to sleep (kse_release()) because there are no other threads to run. Drew, can you try lowering the concurrency? You can either try using pthread_setconcurrency(1) or setting kern.threads.virtual_cpu=1. Julian, why does it take so long for the woken KSE to become active? -- Dan From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 14:23:20 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 680AC16A4CE for ; Mon, 5 Jul 2004 14:23:20 +0000 (GMT) Received: from sccrmhc13.comcast.net (sccrmhc13.comcast.net [204.127.202.64]) by mx1.FreeBSD.org (Postfix) with ESMTP id DB80943D48 for ; Mon, 5 Jul 2004 14:23:19 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc13) with ESMTP id <2004070514231801600o7c0fe>; Mon, 5 Jul 2004 14:23:18 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id HAA46738; Mon, 5 Jul 2004 07:23:16 -0700 (PDT) Date: Mon, 5 Jul 2004 07:23:15 -0700 (PDT) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 14:23:20 -0000 On Mon, 5 Jul 2004, Daniel Eischen wrote: > On Mon, 5 Jul 2004, Julian Elischer wrote: > > > > On Sun, 4 Jul 2004, Andrew Gallatin wrote: > > > > > The problem turned out to be that the worker thread was sleeping in > > > cv_wait_sig() on a cv which was used elsewhere in the driver. When I > > > fixed this, pretty much everything got better. I still don't > > > understand exactly what happened. I have no idea if the worker woke > > > too early, or if the other place this cv was (mis)used was where the > > > early wake happened. (this would be where mx_free() is called). > > > > > > Anyway, it no longer crashes the machine. Sorry to have wasted your > > > time. > > > > > > Now to figure out why libthr does pthread_cond_signal() in my scenario > > > 47us faster than libpthread... (ULE, 1 HTT P4 running SMP kernel) > > > > > > Scenario is that the mainline thread sleeps waiting for a packet > > > (using pthread_cond_timedwait()) and the worker thread is asleep in > > > kernel. When a packet arrives, the worker wakes up, returns from the > > > ioctl, does a pthread_cond_signal() to wakeup the mainline thread, and > > > goes back into the kernel to sleep (via an ioctl). This is the sort > > > of scenario where I thought KSE would be faster than a 1:1 lib.. > > ULE & HTT & KSE is probably the most inefficient combination... > I find true SMP & 4BSD to be better for KSE (perhaps after > julian finishes his cleanup, things will be better). > > > Actually it is the sort of scenario where it is not clear what is best > > ;-) > > > > The mainline thread is asleep in userland. thus to become active, if > > there is no running kernel thread to run it, one needs to be woken up > > from the kernel and made to upcall to userspace to be assigned to run > > the thread. This is pretty much what is happenning in M:M threads. (1:1 > > is in my opinion a non threadded app :-) Except that in KSE there is > > more overhead to do it.. One thing that could be tuned in KSE to make it > > a lot faster is to reverse the order that the threads are run in the > > following way: > > > > ------ currently: > > > > Kernel thread 1 returns from waiting for the packet, and as user thread > > A, signals (enters) the UTS to make user thread B runnable. The UTS > > realises that there is no kernel thread currently running to run thread > > B, and enters the kernel to wake one up, and returns to userland, in > > order to re-enter thread A and go back into teh kernel to wait for the > > next packet. > > Actually, the UTS doesn't really think of it in terms of creating > new kernel threads. It only sees idle and non-idle KSEs. > > > Kernel thread 2 is started up some time after it is awoken and enters > > userland via an upcall and is assigned thread B to run. > > > > This is all assuming there are 2 (or more CPUS). > > If there is only one CPU then when user thread A sleeps in the kernel, > > the kernel thread (1) is allowed to upcall back to the UTS and take on > > user thread B. > > This is true regardless of the number of CPUs, unless you have > system scope threads. Any blocking in the kernel will result > in an upcall. > > > ------ the changed version: > > > > Kernel thread 1 returns from waiting for the packet, and as user thread > > A, signals (enters) the UTS to make user thread B runnable. The UTS > > realises that there is no kernel thread currently running to run thread > > B it enters the kernel briefly to make another thread wake up and > > upcall, and then suspends thread A, and takes on thread B. > > Eventually th e upcall occurs and thread A is assigned to the newly > > awoken thread and re-enters the kernel, waitign for the next kernel. > > > > ------------ > > > > Dan, can the 2nd version happen now? > > can Drew make it happen by assigning a higher priority to thread B? > > No, not yet; pthread_cond_signal() and pthread_mutex_unlock() are > not currently preemption points. I'm not sure that you would want > pthread_cond_signal() to be a preemption point -- I'd imagine that > a lot of applications would do something like this: > > /* sleeping */ > pthread_mutex_lock(&m); > foo->sleeping_count++; > pthread_cond_wait(&cv, &m); > pthread_mutex_unlock(&m); > > /* waking up */ > pthread_mutex_lock(&m); > if (foo->sleeping_count > 0) { > foo->sleeping_count--; > pthread_cond_signal(&cv); > } > pthread_mutex_unlock(&m); > > > If you make pthread_cond_signal() a preemption point, then you > would get some ping-pong-like effect. The newly awoken thread > would block again on the mutex. That may still be faster than going into the kernel and anyway, it would only occur if the other process had a higher priority. and it would DEFINITLY help on a uniprocessor. > > With a small number of threads, it probably doesn't make sense > to have more than one KSE (unless they are mostly CPU-bound). > In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads. > Each time a thread sleeps, the KSE enters the kernel to sleep > (kse_release()) because there are no other threads to run. with 1 KSE it is definitly worth adding pre-emption there.. > > Drew, can you try lowering the concurrency? You can > either try using pthread_setconcurrency(1) or setting > kern.threads.virtual_cpu=1. > > Julian, why does it take so long for the woken KSE to become > active? It will be the shorter of: add up: time to for 'waker' to enter the UTS time for the UTS to decide it can awaken another KSE time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") time until next clock tick on the other processor.. (We STILL do not send an IPI o the other processor yet) to make it schedule a new thread). OR add up: time to for 'waker' to enter the UTS time for the UTS to decide it can awaken another KSE time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") time for tkse_wakeup() to returnn to user land. time for UTS to return to original thread time for thread to so syscall (what, select() ?) and call msleep() time for scheduler to cause an upcall, time for UTS to field upcall time for UTS to schedule the new awoken thread on kernel thread. note, the requested upcall will still happen on the other CPU but the UTS will find no work for it and send it back to sleep with kse_release() again. I suspect that as clock ticks occur at only 10mSEC period, that it will nearly always be the 2nd option. In other words the system is acting as it would with a uniprocessor. compare this with: add up: time to for 'waker' to enter the UTS time for the UTS to decide it can awaken another KSE time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") time for UTS to return to original thread time for UTS to switch to other thread. (*) this is what is being measured here.. however the 'select' thread wil be restarted when the other thread completes and re-does the wait OR a clock tick occurs. julian From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 15:04:39 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5245C16A4CE for ; Mon, 5 Jul 2004 15:04:39 +0000 (GMT) Received: from sccrmhc12.comcast.net (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0E3AB43D55 for ; Mon, 5 Jul 2004 15:04:39 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc12) with ESMTP id <2004070515043701200017kge>; Mon, 5 Jul 2004 15:04:38 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id IAA47237; Mon, 5 Jul 2004 08:04:35 -0700 (PDT) Date: Mon, 5 Jul 2004 08:04:34 -0700 (PDT) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 15:04:39 -0000 On Mon, 5 Jul 2004, Julian Elischer wrote: > > > > I suspect that as clock ticks occur at only 10mSEC period, that it will > nearly always be the 2nd option. In other words the system is acting as > it would with a uniprocessor. > > compare this with: > add up: > > time to for 'waker' to enter the UTS > time for the UTS to decide it can awaken another KSE > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > time for UTS to return to original thread > time for UTS to switch to other thread. > (*) this is what is being measured here.. > > however the 'select' thread wil be restarted when the other thread > completes and re-does the wait OR a clock tick occurs. > > Drew, what happens to the 47uSec if you do: sysctl machdep.cpu_idle_hlt=0 ? > > julian > > > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org" > From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 19:03:31 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0A36416A4CE for ; Mon, 5 Jul 2004 19:03:31 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id B231743D48 for ; Mon, 5 Jul 2004 19:03:30 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i65J3NfP001593; Mon, 5 Jul 2004 15:03:23 -0400 (EDT) Date: Mon, 5 Jul 2004 15:03:23 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 19:03:31 -0000 On Mon, 5 Jul 2004, Julian Elischer wrote: > > > On Mon, 5 Jul 2004, Daniel Eischen wrote: > > > On Mon, 5 Jul 2004, Julian Elischer wrote: > > > > > ------ the changed version: > > > > > > Kernel thread 1 returns from waiting for the packet, and as user thread > > > A, signals (enters) the UTS to make user thread B runnable. The UTS > > > realises that there is no kernel thread currently running to run thread > > > B it enters the kernel briefly to make another thread wake up and > > > upcall, and then suspends thread A, and takes on thread B. > > > Eventually th e upcall occurs and thread A is assigned to the newly > > > awoken thread and re-enters the kernel, waitign for the next kernel. > > > > > > ------------ > > > > > > Dan, can the 2nd version happen now? > > > can Drew make it happen by assigning a higher priority to thread B? > > > > No, not yet; pthread_cond_signal() and pthread_mutex_unlock() are > > not currently preemption points. I'm not sure that you would want > > pthread_cond_signal() to be a preemption point -- I'd imagine that > > a lot of applications would do something like this: > > > > /* sleeping */ > > pthread_mutex_lock(&m); > > foo->sleeping_count++; > > pthread_cond_wait(&cv, &m); > > pthread_mutex_unlock(&m); > > > > /* waking up */ > > pthread_mutex_lock(&m); > > if (foo->sleeping_count > 0) { > > foo->sleeping_count--; > > pthread_cond_signal(&cv); > > } > > pthread_mutex_unlock(&m); > > > > > > If you make pthread_cond_signal() a preemption point, then you > > would get some ping-pong-like effect. The newly awoken thread > > would block again on the mutex. > > That may still be faster than > going into the kernel > > and anyway, it would only occur if the other process had a higher > priority. and it would DEFINITLY help on a uniprocessor. I don't see how it would help on a uniprocessor. On a uniprocessor, as soon as one thread blocks, the other thread will be run. The effect is the same except it lets the original thread run until it blocks. thread A is blocking on ioctl(). thread B is waiting in pthread_cond_wait(). ... time goes by, some event happens and releases thread A... UTS resumes thread A thread A wakes up thread B via pthread_cond_signal() [ thread B is marked runnable and added to UTS run queue ] thread A continues until it blocks [ if A blocks in kernel, upcall is made, else A is blocked in userland and there is no userland <-> kernel boundary crossing. ] UTS runs thread B [ B is resumed in userland since pthread_cond_wait() blocks in userland. ] thread B runs until it blocks Compare this with: thread A is blocking on ioctl(). thread B is waiting in pthread_cond_wait(). ... time goes by, some event happens and releases thread A... UTS resumes thread A thread A wakes up thread B via pthread_cond_signal() [ thread B is marked runnable and added to UTS run queue ] UTS switches to thread B [ thread A is swapped out in userland, thread B is resumed in userland ] thread B runs until it blocks [ if B blocks in kernel, upcall is made, else B is blocked in userland and there is no userland <-> kernel boundary crossing. ] UTS switches to thread A [ thread A is resumed in userland ] thread A continues until it blocks In fact, there is one extra switch in the latter. We could easily add preemption points for higher priority threads. That would allow the application to decide which thread is the most important. The other problem is, for SMP, you want to avoid migrating threads between KSEs (processors). The UTS doesn't currently do this. There is only one scheduling queue per KSE group and all KSEs (allocated for process-scope threads) use the same scheduling queue hung off the group. > > With a small number of threads, it probably doesn't make sense > > to have more than one KSE (unless they are mostly CPU-bound). > > In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads. > > Each time a thread sleeps, the KSE enters the kernel to sleep > > (kse_release()) because there are no other threads to run. > > with 1 KSE it is definitly worth adding pre-emption there.. See above and tell me if you still think so. > > Julian, why does it take so long for the woken KSE to become > > active? > > It will be the shorter of: > > add up: > time to for 'waker' to enter the UTS > time for the UTS to decide it can awaken another KSE > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > time until next clock tick on the other processor.. (We STILL do not > send an IPI o the other processor yet) to make it schedule a new > thread). > > OR > > add up: > time to for 'waker' to enter the UTS > time for the UTS to decide it can awaken another KSE > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > time for tkse_wakeup() to returnn to user land. > time for UTS to return to original thread > time for thread to so syscall (what, select() ?) and call msleep() > time for scheduler to cause an upcall, > time for UTS to field upcall > time for UTS to schedule the new awoken thread on kernel thread. Note that the time for the UTS to decide it can wake another KSE is negligible. At this point the UTS already owns the scheduler lock (no extra locks are needed), and it is merely looping through a linked list of KSEs hung off the group. For SMP there are only 2 KSEs, so it is just a few instructions. > note, the requested upcall will still happen on the other CPU > but the UTS will find no work for it and send it back to sleep with > kse_release() again. > > > > I suspect that as clock ticks occur at only 10mSEC period, that it will > nearly always be the 2nd option. In other words the system is acting as > it would with a uniprocessor. > > compare this with: > add up: > > time to for 'waker' to enter the UTS > time for the UTS to decide it can awaken another KSE > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > time for UTS to return to original thread > time for UTS to switch to other thread. > (*) this is what is being measured here.. > > however the 'select' thread wil be restarted when the other thread > completes and re-does the wait OR a clock tick occurs. Well, since the quoted time was 47usec, the KSE must be getting scheduled before the next clock tick. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 21:44:26 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8DAD716A4CE for ; Mon, 5 Jul 2004 21:44:26 +0000 (GMT) Received: from sccrmhc12.comcast.net (sccrmhc12.comcast.net [204.127.202.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id E85BF43D45 for ; Mon, 5 Jul 2004 21:44:24 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc12) with ESMTP id <200407052144230120001n7ce>; Mon, 5 Jul 2004 21:44:23 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id OAA51974; Mon, 5 Jul 2004 14:44:20 -0700 (PDT) Date: Mon, 5 Jul 2004 14:44:19 -0700 (PDT) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 21:44:26 -0000 On Mon, 5 Jul 2004, Daniel Eischen wrote: > On Mon, 5 Jul 2004, Julian Elischer wrote: > > > > > > > On Mon, 5 Jul 2004, Daniel Eischen wrote: > > > > > On Mon, 5 Jul 2004, Julian Elischer wrote: > > > > > > > ------ the changed version: > > > > > > > > Kernel thread 1 returns from waiting for the packet, and as user thread > > > > A, signals (enters) the UTS to make user thread B runnable. The UTS > > > > realises that there is no kernel thread currently running to run thread > > > > B it enters the kernel briefly to make another thread wake up and > > > > upcall, and then suspends thread A, and takes on thread B. > > > > Eventually th e upcall occurs and thread A is assigned to the newly > > > > awoken thread and re-enters the kernel, waitign for the next kernel. > > > > > > > > ------------ > > > > > > > > Dan, can the 2nd version happen now? > > > > can Drew make it happen by assigning a higher priority to thread B? > > > > > > No, not yet; pthread_cond_signal() and pthread_mutex_unlock() are > > > not currently preemption points. I'm not sure that you would want > > > pthread_cond_signal() to be a preemption point -- I'd imagine that > > > a lot of applications would do something like this: > > > > > > /* sleeping */ > > > pthread_mutex_lock(&m); > > > foo->sleeping_count++; > > > pthread_cond_wait(&cv, &m); > > > pthread_mutex_unlock(&m); > > > > > > /* waking up */ > > > pthread_mutex_lock(&m); > > > if (foo->sleeping_count > 0) { > > > foo->sleeping_count--; > > > pthread_cond_signal(&cv); > > > } > > > pthread_mutex_unlock(&m); > > > > > > > > > If you make pthread_cond_signal() a preemption point, then you > > > would get some ping-pong-like effect. The newly awoken thread > > > would block again on the mutex. > > > > That may still be faster than > > going into the kernel > > > > and anyway, it would only occur if the other process had a higher > > priority. and it would DEFINITLY help on a uniprocessor. > > I don't see how it would help on a uniprocessor. On a > uniprocessor, as soon as one thread blocks, the other > thread will be run. The effect is the same except it > lets the original thread run until it blocks. The problem is that the other thread is NOT blocking for another 40uSecs or so. > > thread A is blocking on ioctl(). > thread B is waiting in pthread_cond_wait(). > > ... time goes by, some event happens and releases thread A... > an upcall goes to the UTS > UTS resumes thread A > thread A wakes up thread B via pthread_cond_signal() > [ thread B is marked runnable and added to UTS run queue ] > thread A continues until it blocks > [ if A blocks in kernel, upcall is made, else A is blocked > in userland and there is no userland <-> kernel boundary > crossing. ] > UTS runs thread B > [ B is resumed in userland since pthread_cond_wait() blocks > in userland. ] > thread B runs until it blocks > > Compare this with: > > thread A is blocking on ioctl(). > thread B is waiting in pthread_cond_wait(). > > ... time goes by, some event happens and releases thread A... > an upcall goes to the UTS > UTS resumes thread A > thread A wakes up thread B via pthread_cond_signal() > [ thread B is marked runnable and added to UTS run queue ] > UTS switches to thread B > [ thread A is swapped out in userland, thread B is resumed > in userland ] > thread B runs until it blocks > [ if B blocks in kernel, upcall is made, else B is blocked > in userland and there is no userland <-> kernel boundary > crossing. ] > UTS switches to thread A > [ thread A is resumed in userland ] > thread A continues until it blocks > > In fact, there is one extra switch in the latter. but that is not the point.. the ioctl is not latency critical.. the worker thread (B) is.. we want B to run as soon as possible.. in scenario A teh following occurs before B runs.. an upcall goes to the UTS > UTS resumes thread A > thread A wakes up thread B via pthread_cond_signal() > [ thread B is marked runnable and added to UTS run queue ] > thread A continues until it blocks > [ if A blocks in kernel, upcall is made, else A is blocked > in userland and there is no userland <-> kernel boundary > crossing. ] > UTS runs thread B > [ B is resumed in userland since pthread_cond_wait() blocks > in userland. ] where in scenario B teh following occurs before B runs: an upcall goes to the UTS > UTS resumes thread A > thread A wakes up thread B via pthread_cond_signal() > [ thread B is marked runnable and added to UTS run queue ] > UTS switches to thread B > [ thread A is swapped out in userland, thread B is resumed > in userland ] which gets B run quicker? A is rerun later, but not neccesarily much later and if we fix the scheduling latency for switching CPUs it will occur quite soon.. > > We could easily add preemption points for higher priority > threads. That would allow the application to decide which > thread is the most important. exactly.. it's the application's decision. > > The other problem is, for SMP, you want to avoid migrating > threads between KSEs (processors). The UTS doesn't currently > do this. There is only one scheduling queue per KSE group > and all KSEs (allocated for process-scope threads) use the > same scheduling queue hung off the group. understood. > > > > With a small number of threads, it probably doesn't make sense > > > to have more than one KSE (unless they are mostly CPU-bound). > > > In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads. > > > Each time a thread sleeps, the KSE enters the kernel to sleep > > > (kse_release()) because there are no other threads to run. > > > > with 1 KSE it is definitly worth adding pre-emption there.. > > See above and tell me if you still think so. definitly. ESPECIALLY if it is not taken when the priorities are the same. i.e the status quo. > > > > Julian, why does it take so long for the woken KSE to become > > > active? > > > > It will be the shorter of: > > > > add up: > > time to for 'waker' to enter the UTS > > time for the UTS to decide it can awaken another KSE > > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > > time until next clock tick on the other processor.. (We STILL do not > > send an IPI o the other processor yet) to make it schedule a new > > thread). > > > > OR > > > > add up: > > time to for 'waker' to enter the UTS > > time for the UTS to decide it can awaken another KSE > > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > > time for tkse_wakeup() to returnn to user land. > > time for UTS to return to original thread > > time for thread to so syscall (what, select() ?) and call msleep() > > time for scheduler to cause an upcall, > > time for UTS to field upcall > > time for UTS to schedule the new awoken thread on kernel thread. > > Note that the time for the UTS to decide it can wake another > KSE is negligible. At this point the UTS already owns the > scheduler lock (no extra locks are needed), and it is merely > looping through a linked list of KSEs hung off the group. > For SMP there are only 2 KSEs, so it is just a few instructions. understood, but the syscall to ask for B to be run on another KSE, (if it doesn't get a chance to run on this one first) doesn't guarantee that the upcall it generates will occur any time soon. > > > note, the requested upcall will still happen on the other CPU > > but the UTS will find no work for it and send it back to sleep with > > kse_release() again. > > > > > > > > I suspect that as clock ticks occur at only 10mSEC period, that it will > > nearly always be the 2nd option. In other words the system is acting as > > it would with a uniprocessor. > > > > compare this with: > > add up: > > > > time to for 'waker' to enter the UTS > > time for the UTS to decide it can awaken another KSE > > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > > time for UTS to return to original thread > > time for UTS to switch to other thread. > > (*) this is what is being measured here.. > > > > however the 'select' thread wil be restarted when the other thread > > completes and re-does the wait OR a clock tick occurs. > > Well, since the quoted time was 47usec, the KSE must be > getting scheduled before the next clock tick. The KSE is not rescheduled, but, rather, continues on with the original thread back into select() again, at which point it converts into an upcall ang goes back to userland where the newly awoken thread is assigned to it and run. taking 47 uSecs. If the userland scheduler just switched to the other thread immediatly, the same thing would happen, except it would be the select() that took place 47 usecs later and not the worker thread. > > -- > Dan Eischen > > From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 22:19:50 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CFDD516A4CE for ; Mon, 5 Jul 2004 22:19:50 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6DAA243D5A for ; Mon, 5 Jul 2004 22:19:50 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i65MJffP007568; Mon, 5 Jul 2004 18:19:42 -0400 (EDT) Date: Mon, 5 Jul 2004 18:19:41 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 22:19:50 -0000 On Mon, 5 Jul 2004, Julian Elischer wrote: > > On Mon, 5 Jul 2004, Daniel Eischen wrote: > > > On Mon, 5 Jul 2004, Julian Elischer wrote: > > > > > > That may still be faster than > > > going into the kernel > > > > > > and anyway, it would only occur if the other process had a higher > > > priority. and it would DEFINITLY help on a uniprocessor. > > > > I don't see how it would help on a uniprocessor. On a > > uniprocessor, as soon as one thread blocks, the other > > thread will be run. The effect is the same except it > > lets the original thread run until it blocks. > > The problem is that the other thread is NOT blocking for another 40uSecs > or so. But libpthread doesn't care which thread runs, as long as _some_ thread runs. You're making assumptions about the application where the library doesn't want to be making any assumptions. > > thread A is blocking on ioctl(). > > thread B is waiting in pthread_cond_wait(). > > > > ... time goes by, some event happens and releases thread A... > > > > an upcall goes to the UTS > > UTS resumes thread A > > thread A wakes up thread B via pthread_cond_signal() > > [ thread B is marked runnable and added to UTS run queue ] > > thread A continues until it blocks > > [ if A blocks in kernel, upcall is made, else A is blocked > > in userland and there is no userland <-> kernel boundary > > crossing. ] > > UTS runs thread B > > [ B is resumed in userland since pthread_cond_wait() blocks > > in userland. ] > > thread B runs until it blocks > > > > Compare this with: > > > > thread A is blocking on ioctl(). > > thread B is waiting in pthread_cond_wait(). > > > > ... time goes by, some event happens and releases thread A... > > > > an upcall goes to the UTS > > UTS resumes thread A > > thread A wakes up thread B via pthread_cond_signal() > > [ thread B is marked runnable and added to UTS run queue ] > > UTS switches to thread B > > [ thread A is swapped out in userland, thread B is resumed > > in userland ] > > thread B runs until it blocks > > [ if B blocks in kernel, upcall is made, else B is blocked > > in userland and there is no userland <-> kernel boundary > > crossing. ] > > UTS switches to thread A > > [ thread A is resumed in userland ] > > thread A continues until it blocks > > > > In fact, there is one extra switch in the latter. > > but that is not the point.. the ioctl is not latency critical.. > the worker thread (B) is.. > we want B to run as soon as possible.. How does the library know that? As I said we can insert preemption points in pthread_cond_signal(), pthread_mutex_unlock(), and perhaps a couple of others, but it would only be for higher priority threads. And I'm not convinced that we even need to do that. > in scenario A teh following occurs before B runs.. > an upcall goes to the UTS > > UTS resumes thread A > > thread A wakes up thread B via pthread_cond_signal() > > [ thread B is marked runnable and added to UTS run queue ] > > thread A continues until it blocks > > [ if A blocks in kernel, upcall is made, else A is blocked > > in userland and there is no userland <-> kernel boundary > > crossing. ] > > UTS runs thread B > > [ B is resumed in userland since pthread_cond_wait() blocks > > in userland. ] > > > where in scenario B teh following occurs before B runs: > an upcall goes to the UTS > > UTS resumes thread A > > thread A wakes up thread B via pthread_cond_signal() > > [ thread B is marked runnable and added to UTS run queue ] > > UTS switches to thread B > > [ thread A is swapped out in userland, thread B is resumed > > in userland ] > > > > which gets B run quicker? > > A is rerun later, but not neccesarily much later and if we fix the > scheduling latency for switching CPUs it will occur quite soon.. > > > > > > We could easily add preemption points for higher priority > > threads. That would allow the application to decide which > > thread is the most important. > > exactly.. it's the application's decision. > > > > > The other problem is, for SMP, you want to avoid migrating > > threads between KSEs (processors). The UTS doesn't currently > > do this. There is only one scheduling queue per KSE group > > and all KSEs (allocated for process-scope threads) use the > > same scheduling queue hung off the group. > > understood. > > > > > > > With a small number of threads, it probably doesn't make sense > > > > to have more than one KSE (unless they are mostly CPU-bound). > > > > In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads. > > > > Each time a thread sleeps, the KSE enters the kernel to sleep > > > > (kse_release()) because there are no other threads to run. > > > > > > with 1 KSE it is definitly worth adding pre-emption there.. > > > > See above and tell me if you still think so. > > definitly. ESPECIALLY if it is not taken when the priorities are the > same. i.e the status quo. > > > > compare this with: > > > add up: > > > > > > time to for 'waker' to enter the UTS > > > time for the UTS to decide it can awaken another KSE > > > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > > > time for UTS to return to original thread > > > time for UTS to switch to other thread. > > > (*) this is what is being measured here.. > > > > > > however the 'select' thread wil be restarted when the other thread > > > completes and re-does the wait OR a clock tick occurs. > > > > Well, since the quoted time was 47usec, the KSE must be > > getting scheduled before the next clock tick. > > The KSE is not rescheduled, but, rather, continues on with the original > thread back into select() again, at which point it converts into an > upcall ang goes back to userland where the newly awoken thread is > assigned to it and run. taking 47 uSecs. > > If the userland scheduler just switched to the other thread immediatly, > the same thing would happen, except it would be the select() that took > place 47 usecs later and not the worker thread. I don't think you can assume the worker thread is being run on the same KSE. HTT is enabled and there are two KSEs. Look at it this way. The thread waiting in ioctl() (call it the I/O thread) is going continuously: while (1) { /* Wait for some event. */ ioctl(...); /* Signal the worker thread. */ pthread_cond_signal(&cv); } and the worker thread is going continuously: while (1) { /* Wait for events. */ pthread_mutex_lock(&m); pthread_cond_wait(&cv, &m); pthread_mutex_unlock(&m); /* Process event. */ ... } It doesn't matter which thread gets run after the I/O thread calls pthread_cond_signal(). Either way, the I/O thread has to reenter the kernel and wait for the next event. And that time adds to the latency of the worker thread the _next_ time. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 22:44:18 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6E3F316A4CE for ; Mon, 5 Jul 2004 22:44:18 +0000 (GMT) Received: from rwcrmhc11.comcast.net (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5A30943D31 for ; Mon, 5 Jul 2004 22:44:18 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc11) with ESMTP id <2004070522441601300j9h20e>; Mon, 5 Jul 2004 22:44:17 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id PAA52659; Mon, 5 Jul 2004 15:44:15 -0700 (PDT) Date: Mon, 5 Jul 2004 15:44:14 -0700 (PDT) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 22:44:18 -0000 On Mon, 5 Jul 2004, Daniel Eischen wrote: > On Mon, 5 Jul 2004, Julian Elischer wrote: > > > > On Mon, 5 Jul 2004, Daniel Eischen wrote: > > > > > On Mon, 5 Jul 2004, Julian Elischer wrote: > > > > > > > > That may still be faster than > > > > going into the kernel > > > > > > > > and anyway, it would only occur if the other process had a higher > > > > priority. and it would DEFINITLY help on a uniprocessor. > > > > > > I don't see how it would help on a uniprocessor. On a > > > uniprocessor, as soon as one thread blocks, the other > > > thread will be run. The effect is the same except it > > > lets the original thread run until it blocks. > > > > The problem is that the other thread is NOT blocking for another 40uSecs > > or so. > > But libpthread doesn't care which thread runs, as long as _some_ > thread runs. You're making assumptions about the application > where the library doesn't want to be making any assumptions. No, I'm allowing the app to make the decision. > > > > thread A is blocking on ioctl(). > > > thread B is waiting in pthread_cond_wait(). > > > > > > ... time goes by, some event happens and releases thread A... > > > > > > > an upcall goes to the UTS > > > UTS resumes thread A > > > thread A wakes up thread B via pthread_cond_signal() > > > [ thread B is marked runnable and added to UTS run queue ] > > > thread A continues until it blocks > > > [ if A blocks in kernel, upcall is made, else A is blocked > > > in userland and there is no userland <-> kernel boundary > > > crossing. ] > > > UTS runs thread B > > > [ B is resumed in userland since pthread_cond_wait() blocks > > > in userland. ] > > > thread B runs until it blocks > > > > > > Compare this with: > > > > > > thread A is blocking on ioctl(). > > > thread B is waiting in pthread_cond_wait(). > > > > > > ... time goes by, some event happens and releases thread A... > > > > > > > an upcall goes to the UTS > > > UTS resumes thread A > > > thread A wakes up thread B via pthread_cond_signal() > > > [ thread B is marked runnable and added to UTS run queue ] > > > UTS switches to thread B > > > [ thread A is swapped out in userland, thread B is resumed > > > in userland ] > > > thread B runs until it blocks > > > [ if B blocks in kernel, upcall is made, else B is blocked > > > in userland and there is no userland <-> kernel boundary > > > crossing. ] > > > UTS switches to thread A > > > [ thread A is resumed in userland ] > > > thread A continues until it blocks > > > > > > In fact, there is one extra switch in the latter. > > > > but that is not the point.. the ioctl is not latency critical.. > > the worker thread (B) is.. > > we want B to run as soon as possible.. > > How does the library know that? As I said we can insert > preemption points in pthread_cond_signal(), pthread_mutex_unlock(), > and perhaps a couple of others, but it would only be for higher > priority threads. And I'm not convinced that we even need to > do that. let the app set priorities.. that's wat they are for. > > > in scenario A teh following occurs before B runs.. > > an upcall goes to the UTS > > > UTS resumes thread A > > > thread A wakes up thread B via pthread_cond_signal() > > > [ thread B is marked runnable and added to UTS run queue ] > > > thread A continues until it blocks > > > [ if A blocks in kernel, upcall is made, else A is blocked > > > in userland and there is no userland <-> kernel boundary > > > crossing. ] > > > UTS runs thread B > > > [ B is resumed in userland since pthread_cond_wait() blocks > > > in userland. ] > > > > > > where in scenario B teh following occurs before B runs: > > an upcall goes to the UTS > > > UTS resumes thread A > > > thread A wakes up thread B via pthread_cond_signal() > > > [ thread B is marked runnable and added to UTS run queue ] > > > UTS switches to thread B > > > [ thread A is swapped out in userland, thread B is resumed > > > in userland ] > > > > > > > > which gets B run quicker? > > > > A is rerun later, but not neccesarily much later and if we fix the > > scheduling latency for switching CPUs it will occur quite soon.. > > > > > > > > > > We could easily add preemption points for higher priority > > > threads. That would allow the application to decide which > > > thread is the most important. > > > > exactly.. it's the application's decision. > > > > > > > > The other problem is, for SMP, you want to avoid migrating > > > threads between KSEs (processors). The UTS doesn't currently > > > do this. There is only one scheduling queue per KSE group > > > and all KSEs (allocated for process-scope threads) use the > > > same scheduling queue hung off the group. > > > > understood. > > > > > > > > > > With a small number of threads, it probably doesn't make sense > > > > > to have more than one KSE (unless they are mostly CPU-bound). > > > > > In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads. > > > > > Each time a thread sleeps, the KSE enters the kernel to sleep > > > > > (kse_release()) because there are no other threads to run. > > > > > > > > with 1 KSE it is definitly worth adding pre-emption there.. > > > > > > See above and tell me if you still think so. > > > > definitly. ESPECIALLY if it is not taken when the priorities are the > > same. i.e the status quo. > > > > > > compare this with: > > > > add up: > > > > > > > > time to for 'waker' to enter the UTS > > > > time for the UTS to decide it can awaken another KSE > > > > time for the UTS to call kse_wakeup() (from kerne entry to "wakeup()") > > > > time for UTS to return to original thread > > > > time for UTS to switch to other thread. > > > > (*) this is what is being measured here.. > > > > > > > > however the 'select' thread wil be restarted when the other thread > > > > completes and re-does the wait OR a clock tick occurs. > > > > > > Well, since the quoted time was 47usec, the KSE must be > > > getting scheduled before the next clock tick. > > > > The KSE is not rescheduled, but, rather, continues on with the original > > thread back into select() again, at which point it converts into an > > upcall ang goes back to userland where the newly awoken thread is > > assigned to it and run. taking 47 uSecs. > > > > If the userland scheduler just switched to the other thread immediatly, > > the same thing would happen, except it would be the select() that took > > place 47 usecs later and not the worker thread. > > I don't think you can assume the worker thread is being run > on the same KSE. HTT is enabled and there are two KSEs. it probably is, but either way it would be quicker. > > Look at it this way. The thread waiting in ioctl() (call it > the I/O thread) is going continuously: > > while (1) { > /* Wait for some event. */ > ioctl(...); > > /* Signal the worker thread. */ > pthread_cond_signal(&cv); > } > > and the worker thread is going continuously: > > while (1) { > /* Wait for events. */ > pthread_mutex_lock(&m); > pthread_cond_wait(&cv, &m); > pthread_mutex_unlock(&m); > /* Process event. */ > ... > } > > It doesn't matter which thread gets run after the I/O thread > calls pthread_cond_signal(). Either way, the I/O thread has > to reenter the kernel and wait for the next event. And that > time adds to the latency of the worker thread the _next_ time. not assuming that the worker thread tries to see if there is more work before it yields. All I'm saying is that the app should decide the priority. and you agree that it would be quicker in the case of the 1KSE case? > > -- > Dan Eischen > > From owner-freebsd-threads@FreeBSD.ORG Mon Jul 5 23:55:48 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E149016A4CE for ; Mon, 5 Jul 2004 23:55:48 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9230C43D31 for ; Mon, 5 Jul 2004 23:55:48 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i65NtffP025039; Mon, 5 Jul 2004 19:55:41 -0400 (EDT) Date: Mon, 5 Jul 2004 19:55:41 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Jul 2004 23:55:49 -0000 On Mon, 5 Jul 2004, Julian Elischer wrote: > On Mon, 5 Jul 2004, Daniel Eischen wrote: > > > > It doesn't matter which thread gets run after the I/O thread > > calls pthread_cond_signal(). Either way, the I/O thread has > > to reenter the kernel and wait for the next event. And that > > time adds to the latency of the worker thread the _next_ time. > > not assuming that the worker thread tries to see if there is more work > before it yields. > > All I'm saying is that the app should decide the priority. > and you agree that it would be quicker in the case of the > 1KSE case? Latency is reduced, but overall throughput(*) is also reduced because of the extra thread switch for every wakeup. (*) Assuming that events occur fast enough. I'll have a look at adding preemption points, but I still have my doubts that it really matters... -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 15:15:05 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7078016A4CE for ; Tue, 6 Jul 2004 15:15:05 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 04E1A43D3F for ; Tue, 6 Jul 2004 15:15:05 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i66FEtqM007991 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jul 2004 11:14:55 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i66FEnKg017484; Tue, 6 Jul 2004 11:14:49 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16618.49641.241488.96595@grasshopper.cs.duke.edu> Date: Tue, 6 Jul 2004 11:14:49 -0400 (EDT) To: Julian Elischer In-Reply-To: References: X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 15:15:05 -0000 Julian Elischer writes: > > Drew, what happens to the 47uSec if you do: > sysctl machdep.cpu_idle_hlt=0 > The gap between libthr and libpthread remains constant, but both get at 10us speedup. Drew From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 15:51:39 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1374B16A4CE for ; Tue, 6 Jul 2004 15:51:39 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id C683943D41 for ; Tue, 6 Jul 2004 15:51:38 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i66FpZqM013468 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jul 2004 11:51:35 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i66FpUk6017516; Tue, 6 Jul 2004 11:51:30 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16618.51842.291266.840333@grasshopper.cs.duke.edu> Date: Tue, 6 Jul 2004 11:51:30 -0400 (EDT) To: Daniel Eischen In-Reply-To: References: X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: Julian Elischer cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 15:51:39 -0000 Daniel Eischen writes: > > With a small number of threads, it probably doesn't make sense > to have more than one KSE (unless they are mostly CPU-bound). > In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads. Actually, there are 2 procs, each with 2 threads. To elaborate, there are two processes, each running one instance of a pingpong test for each of 2 NICs in the box. > Each time a thread sleeps, the KSE enters the kernel to sleep > (kse_release()) because there are no other threads to run. > > Drew, can you try lowering the concurrency? You can > either try using pthread_setconcurrency(1) or setting > kern.threads.virtual_cpu=1. Here is the raw data for a ping-pong between 2 nics on the same box with various configurations of cpu_idle_hlt and virtual_cpu. Polling 5.0us Linux 20.4us linulator 82.5us thr 75.1us kse 123.6us machdep.cpu_idle_hlt=0 linulator 74.0us thr 64.6us kse 111.9 kern.threads.virtual_cpu=1 machdep.cpu_idle_hlt=0 kse 100.8 kern.threads.virtual_cpu=1 machdep.cpu_idle_hlt=0 kse 93.9 Key: Linux: kernel 2.66, Debian libc6 2.2.5-11.5 (no sysenter, or NPTL) linulator: static binary used above, run under COMPAT_LINUX thr: libthr (via libmap.conf) kse: libpthread BTW, there's a little more fat in the interrupt/wakeup path in the FreeBSD MD driver code (condvars, MPSAFE intr, rather than an INTR_FAST and some homebrew technique). Based on other measurements, I don't think its 54us worth of fat (more like 15 or 20usec). Drew From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 16:05:16 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5CA1516A4CE for ; Tue, 6 Jul 2004 16:05:16 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id F0E9D43D45 for ; Tue, 6 Jul 2004 16:05:15 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i66G5DqM015826 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jul 2004 12:05:13 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i66G58ME017536; Tue, 6 Jul 2004 12:05:08 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16618.52660.343754.933037@grasshopper.cs.duke.edu> Date: Tue, 6 Jul 2004 12:05:08 -0400 (EDT) To: Julian Elischer In-Reply-To: References: X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 16:05:16 -0000 FWIW, inserting a pthread_yield() just before the ioctl call in the worker thread speeds things up quite a bit (100us -> 73us) in combination with kern.threads.virtual_cpu=1. Drew From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 17:20:06 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 031B916A4CE for ; Tue, 6 Jul 2004 17:20:06 +0000 (GMT) Received: from tabby.sonn.com (tabby.sonn.com [193.72.216.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id EBB1F43D58 for ; Tue, 6 Jul 2004 17:20:05 +0000 (GMT) (envelope-from gersh@tabby.sonn.com) Received: from tabby.sonn.com (gersh@localhost [127.0.0.1]) by tabby.sonn.com (8.12.9/8.12.3) with ESMTP id i66HOfDE063579 for ; Tue, 6 Jul 2004 10:24:41 -0700 (PDT) (envelope-from gersh@tabby.sonn.com) Received: from localhost (gersh@localhost) by tabby.sonn.com (8.12.9/8.12.3/Submit) with ESMTP id i66HOfUd008262 for ; Tue, 6 Jul 2004 10:24:41 -0700 (PDT) Date: Tue, 6 Jul 2004 10:24:41 -0700 (PDT) From: Logan Gabriel To: freebsd-threads@freebsd.org Message-ID: <20040706093353.K13095@tabby.sonn.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Signal handling thread not working. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 17:20:06 -0000 Hello, I am writing a multi threaded daemon for a number of unix platforms. In this daemon I create one thread to do signal handling with sigwait(). Signals in all other threads are blocked out. There is a problem with SIGCHLD in many FreeBSD thread libs, (It creates a zombie process). (Source code for example application: http://67.41.131.100/pthreadbug.c) The real application runs on 4.x and uses linuxthreads from ports so that is what I am most interested in having resolved, however I have noticed this code produces zombie processes with the normal user threads library (-pthread) on 4.x and with -lc_r in 5.x # # FreeBSD 4.9-STABLE #0: Wed Nov 12 23:16:46 GMT 2003 # [root@vmware] gcc -o pthreadbug pthreadbug.c -pthread ./pthreadbug ### Sigwait is now blocking. ### Child process 33108 exited. ### Child process 33109 exited. ^Z [1]+ Stopped ./pthreadbug [root@vmware] ps -axwwg | grep pthreadbug 33107 p1 T 0:00.01 ./pthreadbug 33108 p1 Z 0:00.00 (pthreadbug) 33109 p1 Z 0:00.00 (pthreadbug) Compilation with the linux threads library: gcc -I/usr/local/include/pthread/linuxthreads -o pthreadbug pthreadbug.c -L/usr/local/lib -llthread In a very recent version of -CURRENT (two weeks ago) -lpthread works as expected, however -lc_r does not. This exact same code produces the expected output (no zombie procs) in Linux (RedHat 9.1 / SuSE 9.1), and Solaris 8, so I am fairly confident this code "should" work as I am expecting. Any thoughts? From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 17:42:24 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 83FB116A4CE for ; Tue, 6 Jul 2004 17:42:24 +0000 (GMT) Received: from sccrmhc11.comcast.net (sccrmhc11.comcast.net [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id D88A843D4C for ; Tue, 6 Jul 2004 17:42:23 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc11) with ESMTP id <20040706174222011009pojde>; Tue, 6 Jul 2004 17:42:23 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id KAA64674; Tue, 6 Jul 2004 10:42:20 -0700 (PDT) Date: Tue, 6 Jul 2004 10:42:18 -0700 (PDT) From: Julian Elischer To: Andrew Gallatin In-Reply-To: <16618.51842.291266.840333@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 17:42:24 -0000 On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > Daniel Eischen writes: > > > > With a small number of threads, it probably doesn't make sense > > to have more than one KSE (unless they are mostly CPU-bound). > > In Drew's example, there are 2 KSEs (HTT enabled) and only 2 threads. > > Actually, there are 2 procs, each with 2 threads. > > To elaborate, there are two processes, each running one instance of a > pingpong test for each of 2 NICs in the box. > > > Each time a thread sleeps, the KSE enters the kernel to sleep > > (kse_release()) because there are no other threads to run. > > > > Drew, can you try lowering the concurrency? You can > > either try using pthread_setconcurrency(1) or setting > > kern.threads.virtual_cpu=1. > > > Here is the raw data for a ping-pong between 2 nics on the same box > with various configurations of cpu_idle_hlt and virtual_cpu. tried libc_r? > > > > Polling 5.0us > Linux 20.4us > linulator 82.5us > thr 75.1us > kse 123.6us > > machdep.cpu_idle_hlt=0 > linulator 74.0us > thr 64.6us > kse 111.9 > > kern.threads.virtual_cpu=1 > machdep.cpu_idle_hlt=0 > kse 100.8 > > kern.threads.virtual_cpu=1 > machdep.cpu_idle_hlt=0 > kse 93.9 > > > Key: > Linux: kernel 2.66, Debian libc6 2.2.5-11.5 (no sysenter, or NPTL) > linulator: static binary used above, run under COMPAT_LINUX > thr: libthr (via libmap.conf) > kse: libpthread > > > BTW, there's a little more fat in the interrupt/wakeup path in the > FreeBSD MD driver code (condvars, MPSAFE intr, rather than an > INTR_FAST and some homebrew technique). Based on other measurements, I > don't think its 54us worth of fat (more like 15 or 20usec). > > Drew > From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 17:47:07 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 36FF716A4DD for ; Tue, 6 Jul 2004 17:47:06 +0000 (GMT) Received: from rwcrmhc11.comcast.net (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id BB84843D1F for ; Tue, 6 Jul 2004 17:47:06 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc11) with ESMTP id <2004070617470501300jdg6ee>; Tue, 6 Jul 2004 17:47:06 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id KAA64731; Tue, 6 Jul 2004 10:47:04 -0700 (PDT) Date: Tue, 6 Jul 2004 10:47:02 -0700 (PDT) From: Julian Elischer To: Andrew Gallatin In-Reply-To: <16618.52660.343754.933037@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 17:47:09 -0000 On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > FWIW, inserting a pthread_yield() just before the ioctl call in the > worker thread speeds things up quite a bit (100us -> 73us) in > combination with kern.threads.virtual_cpu=1. what about with kern.threads.virtual_cpu untouched? and what about with the hlt sysctl? does your worker thread loop to check if there is more work before waiting to be notified? > > Drew > From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 18:19:37 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 442A616A4CE; Tue, 6 Jul 2004 18:19:37 +0000 (GMT) Received: from rwcrmhc11.comcast.net (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2662743D5E; Tue, 6 Jul 2004 18:19:37 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc11) with ESMTP id <2004070618193601300jbt5he>; Tue, 6 Jul 2004 18:19:36 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id LAA65124; Tue, 6 Jul 2004 11:19:34 -0700 (PDT) Date: Tue, 6 Jul 2004 11:19:33 -0700 (PDT) From: Julian Elischer To: Andrew Gallatin In-Reply-To: <16618.49641.241488.96595@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: John Baldwin Subject: Activating idle threads. X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 18:19:37 -0000 On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > Julian Elischer writes: > > > > Drew, what happens to the 47uSec if you do: > > sysctl machdep.cpu_idle_hlt=0 > > > > The gap between libthr and libpthread remains constant, but > both get at 10us speedup. ok, then I had a patch for 4.x that could help if updated to -current. http://www.freebsd.org/~julian/it.diff the idea is that when we schedule a process, we kick an idle cpu if there is one. That makes it take on the newly runnable thread instead of waiting until the next interrupt or tick. I think that jhb is probably the person to actually bring this concept up-to-date as he has his fingers in there at the moment.. I'll cc him.. julian From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 18:26:57 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0716C16A4CE for ; Tue, 6 Jul 2004 18:26:57 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 887CA43D39 for ; Tue, 6 Jul 2004 18:26:56 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i66IQifP015923; Tue, 6 Jul 2004 14:26:44 -0400 (EDT) Date: Tue, 6 Jul 2004 14:26:44 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Andrew Gallatin In-Reply-To: <16618.52660.343754.933037@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Julian Elischer cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 18:26:57 -0000 On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > FWIW, inserting a pthread_yield() just before the ioctl call in the > worker thread speeds things up quite a bit (100us -> 73us) in > combination with kern.threads.virtual_cpu=1. Then I guess preemption and setting thread priorities accordingly would help your problem... -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 19:40:13 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7C11C16A4CE for ; Tue, 6 Jul 2004 19:40:13 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2993E43D39 for ; Tue, 6 Jul 2004 19:40:13 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i66Je7qM010144 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jul 2004 15:40:07 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i66Je2ho017738; Tue, 6 Jul 2004 15:40:02 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16619.18.84603.913269@grasshopper.cs.duke.edu> Date: Tue, 6 Jul 2004 15:40:02 -0400 (EDT) To: Julian Elischer In-Reply-To: References: <16618.51842.291266.840333@grasshopper.cs.duke.edu> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 19:40:13 -0000 Julian Elischer writes: > > tried libc_r? > It can't work there -- As soon as the worker thread goes to sleep in the driver ioctl, everything comes to a screaching halt :-( Drew From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 20:14:34 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3777516A4CE for ; Tue, 6 Jul 2004 20:14:34 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id BDFC043D31 for ; Tue, 6 Jul 2004 20:14:33 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i66KEVqM013859 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jul 2004 16:14:31 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i66KEQsX017769; Tue, 6 Jul 2004 16:14:26 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16619.2082.399880.649617@grasshopper.cs.duke.edu> Date: Tue, 6 Jul 2004 16:14:26 -0400 (EDT) To: Julian Elischer In-Reply-To: References: <16618.52660.343754.933037@grasshopper.cs.duke.edu> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 20:14:34 -0000 Julian Elischer writes: > > > On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > > > > FWIW, inserting a pthread_yield() just before the ioctl call in the > > worker thread speeds things up quite a bit (100us -> 73us) in > > combination with kern.threads.virtual_cpu=1. > > what about with kern.threads.virtual_cpu untouched? > and what about with the hlt sysctl? kern.threads.virtual_cpu=2 machdep.cpu_idle_hlt=1 no yeild 123.6us yeild 116.8us kern.threads.virtual_cpu=2 machdep.cpu_idle_hlt=0 no yield 111.9 yield 112.9 kern.threads.virtual_cpu=1 machdep.cpu_idle_hlt=1 no yeild 100.8 yeild 75.0 kern.threads.virtual_cpu=1 machdep.cpu_idle_hlt=0 no yield 93.9 ield 67.9 > does your worker thread loop to check if there is more work before > waiting to be notified? Yes. He takes a mutex, loops over all completed events, sending pthread_signals as required, then releases the mutex and sleeps via an ioctl. Drew From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 20:22:41 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EB15716A4CE for ; Tue, 6 Jul 2004 20:22:41 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id A2D0E43D55 for ; Tue, 6 Jul 2004 20:22:41 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i66KMYfP010689; Tue, 6 Jul 2004 16:22:34 -0400 (EDT) Date: Tue, 6 Jul 2004 16:22:34 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Andrew Gallatin In-Reply-To: <16619.2082.399880.649617@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Julian Elischer cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 20:22:42 -0000 On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > Julian Elischer writes: > > > > > > On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > > > > > > > FWIW, inserting a pthread_yield() just before the ioctl call in the > > > worker thread speeds things up quite a bit (100us -> 73us) in > > > combination with kern.threads.virtual_cpu=1. > > > > what about with kern.threads.virtual_cpu untouched? > > and what about with the hlt sysctl? > > > kern.threads.virtual_cpu=2 > machdep.cpu_idle_hlt=1 > > no yeild 123.6us > yeild 116.8us > > kern.threads.virtual_cpu=2 > machdep.cpu_idle_hlt=0 > no yield 111.9 > yield 112.9 > > kern.threads.virtual_cpu=1 > machdep.cpu_idle_hlt=1 > no yeild 100.8 > yeild 75.0 > > kern.threads.virtual_cpu=1 > machdep.cpu_idle_hlt=0 > no yield 93.9 > ield 67.9 > > > > does your worker thread loop to check if there is more work before > > waiting to be notified? > > Yes. He takes a mutex, loops over all completed events, sending > pthread_signals as required, then releases the mutex and sleeps via > an ioctl. Note that he is holding the mutex while calling pthread_cond_signal(). If we enable preemption in pthread_cond_signal(), then I suspect it would be even worse than without preemption. I think the only place where it is sane to enable preemption is on pthread_mutex_unlock(). -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 20:30:28 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A167516A4CE for ; Tue, 6 Jul 2004 20:30:28 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3418B43D5C for ; Tue, 6 Jul 2004 20:30:28 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i66KUPqM015699 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jul 2004 16:30:25 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i66KUJwo017788; Tue, 6 Jul 2004 16:30:19 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16619.3035.165728.569632@grasshopper.cs.duke.edu> Date: Tue, 6 Jul 2004 16:30:19 -0400 (EDT) To: Daniel Eischen In-Reply-To: References: <16619.2082.399880.649617@grasshopper.cs.duke.edu> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: Julian Elischer cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 20:30:28 -0000 Daniel Eischen writes: > Note that he is holding the mutex while calling pthread_cond_signal(). > If we enable preemption in pthread_cond_signal(), then I suspect it > would be even worse than without preemption. > > I think the only place where it is sane to enable preemption is > on pthread_mutex_unlock(). That certainly makes sense to me. But I don't want to see you optimize the threads lib just for my app. I can always put in an #ifdef FreeBSD pthread_yield() #endif But what I'm still wondering is where the ~3x slowdown (20usec linux -> ~60usec, libthr) is coming from. Is this a factor of the context switch time and ithreads? I need to move the driver to a fast interrupt and I need to test SCHED_BSD as well. I should probably cvs update to get jhb's scheduling changes. Drew From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 20:48:39 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C73016A4CE for ; Tue, 6 Jul 2004 20:48:39 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id D9CFF43D46 for ; Tue, 6 Jul 2004 20:48:38 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i66KmTfP016121; Tue, 6 Jul 2004 16:48:31 -0400 (EDT) Date: Tue, 6 Jul 2004 16:48:29 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Andrew Gallatin In-Reply-To: <16619.3035.165728.569632@grasshopper.cs.duke.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Julian Elischer cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 20:48:39 -0000 On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > Daniel Eischen writes: > > Note that he is holding the mutex while calling pthread_cond_signal(). > > If we enable preemption in pthread_cond_signal(), then I suspect it > > would be even worse than without preemption. > > > > I think the only place where it is sane to enable preemption is > > on pthread_mutex_unlock(). > > That certainly makes sense to me. But I don't want to see you > optimize the threads lib just for my app. I can always put in an > #ifdef FreeBSD > pthread_yield() > #endif > > But what I'm still wondering is where the ~3x slowdown (20usec linux > -> ~60usec, libthr) is coming from. Is this a factor of the context > switch time and ithreads? I suspect our kernel is butt-slow compared with Linux in general. Mysql under native linux is similarly faster. > I need to move the driver to a fast interrupt and I need to test > SCHED_BSD as well. I should probably cvs update to get jhb's > scheduling changes. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 21:04:35 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4FE2916A4CE for ; Tue, 6 Jul 2004 21:04:35 +0000 (GMT) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0F4B743D3F for ; Tue, 6 Jul 2004 21:04:35 +0000 (GMT) (envelope-from gallatin@cs.duke.edu) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.12.10/8.12.10) with ESMTP id i66L4XqM020584 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Jul 2004 17:04:33 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.12.9p2/8.12.9/Submit) id i66L4R7F017816; Tue, 6 Jul 2004 17:04:27 -0400 (EDT) (envelope-from gallatin) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <16619.5083.714960.598268@grasshopper.cs.duke.edu> Date: Tue, 6 Jul 2004 17:04:27 -0400 (EDT) To: Daniel Eischen In-Reply-To: References: <16619.3035.165728.569632@grasshopper.cs.duke.edu> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid cc: Julian Elischer cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 21:04:35 -0000 Daniel Eischen writes: > On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > > > > Daniel Eischen writes: > > > Note that he is holding the mutex while calling pthread_cond_signal(). > > > If we enable preemption in pthread_cond_signal(), then I suspect it > > > would be even worse than without preemption. > > > > > > I think the only place where it is sane to enable preemption is > > > on pthread_mutex_unlock(). > > > > That certainly makes sense to me. But I don't want to see you > > optimize the threads lib just for my app. I can always put in an > > #ifdef FreeBSD > > pthread_yield() > > #endif > > > > But what I'm still wondering is where the ~3x slowdown (20usec linux > > -> ~60usec, libthr) is coming from. Is this a factor of the context > > switch time and ithreads? > > I suspect our kernel is butt-slow compared with Linux in general. > Mysql under native linux is similarly faster. The butt-slowness is why I always whine about performance :-( > > I need to move the driver to a fast interrupt and I need to test > > SCHED_BSD as well. I should probably cvs update to get jhb's > > scheduling changes. With jhb's preemption changes, SCHED_4BSD, machdep.cpu_idle_hlt=0, kern.threads.virtual_cpu=1, pthread_yield(), and a partridge in a pear tree, I can get it down to 48us with KSE. Best for libthr is now 57us. This is apples to oranges, because of the ULE/BSD and preemption changes. So I'll wait until ULE works before trying to see if the change was preemption.. Drew From owner-freebsd-threads@FreeBSD.ORG Tue Jul 6 21:22:24 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 805C016A4CE for ; Tue, 6 Jul 2004 21:22:24 +0000 (GMT) Received: from sccrmhc11.comcast.net (sccrmhc11.comcast.net [204.127.202.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9CFF643D48 for ; Tue, 6 Jul 2004 21:22:23 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (sccrmhc11) with ESMTP id <2004070621222201100i7lkae>; Tue, 6 Jul 2004 21:22:22 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id OAA67182; Tue, 6 Jul 2004 14:22:16 -0700 (PDT) Date: Tue, 6 Jul 2004 14:22:15 -0700 (PDT) From: Julian Elischer To: Daniel Eischen In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Andrew Gallatin cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2004 21:22:24 -0000 On Tue, 6 Jul 2004, Daniel Eischen wrote: > On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > > > > Daniel Eischen writes: > > > Note that he is holding the mutex while calling pthread_cond_signal(). > > > If we enable preemption in pthread_cond_signal(), then I suspect it > > > would be even worse than without preemption. > > > > > > I think the only place where it is sane to enable preemption is > > > on pthread_mutex_unlock(). > > > > That certainly makes sense to me. But I don't want to see you > > optimize the threads lib just for my app. I can always put in an > > #ifdef FreeBSD > > pthread_yield() > > #endif > > > > But what I'm still wondering is where the ~3x slowdown (20usec linux > > -> ~60usec, libthr) is coming from. Is this a factor of the context > > switch time and ithreads? > > I suspect our kernel is butt-slow compared with Linux in general. > Mysql under native linux is similarly faster. It'd be interesting to see a kernel compiled with config -p and the result of the profiling.. (when nothing else is running.) I presume all teh debugging is turned off. > > > I need to move the driver to a fast interrupt and I need to test > > SCHED_BSD as well. I should probably cvs update to get jhb's > > scheduling changes. > > -- > Dan Eischen > > From owner-freebsd-threads@FreeBSD.ORG Wed Jul 7 08:57:41 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3A02B16A4CE for ; Wed, 7 Jul 2004 08:57:41 +0000 (GMT) Received: from anor.ics.muni.cz (anor.ics.muni.cz [147.251.4.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 586AB43D1F for ; Wed, 7 Jul 2004 08:57:40 +0000 (GMT) (envelope-from hopet@ics.muni.cz) Received: from KLOBOUCEK (kloboucek.ics.muni.cz [147.251.3.38]) (user=hopet@META mech=LOGIN bits=0) by anor.ics.muni.cz (8.12.1/8.12.1) with ESMTP id i678vbnD026309 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO) for ; Wed, 7 Jul 2004 10:57:38 +0200 From: "Petr Holub" To: Date: Wed, 7 Jul 2004 10:57:36 +0200 Message-ID: <000d01c46400$732b6d60$2603fb93@KLOBOUCEK> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2377.0 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 X-Muni-Spam-TestIP: 147.251.3.38 X-Muni-Virus-Test: Clean Subject: sem_wait as cancellation point X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2004 08:57:41 -0000 Hi all, I wonder why sem_wait is not a pthread cancellation point on FreeBSD-5. According to Linux man page: Cancellation points are those points in the program execution  where  a test for pending cancellation requests is performed and cancellation is executed if positive. The following POSIX threads functions are cancellation points:    pthread_join (3)    pthread_cond_wait (3)    pthread_cond_timedwait (3)    pthread_testcancel (3)    sem_wait (3)    sigwait (3) Currently, I'm using a workaround with sem_trywait and usleep, but it's not as efficient as sem_wait being a cancellation point. Thanks a lot, Petr PS: please keep me in Cc: as I'm not member of this list. ================================================================ Petr Holub CESNET z.s.p.o. Supercomputing Center Brno Zikova 4 Institute of Compt. Science 162 00 Praha 6, CZ Masaryk University Czech Republic Botanicka 68a, 60200 Brno, CZ e-mail: Petr.Holub@cesnet.cz phone: +420-549493944 fax: +420-541212747 e-mail: hopet@ics.muni.cz From owner-freebsd-threads@FreeBSD.ORG Wed Jul 7 10:42:55 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 43D5516A4CE for ; Wed, 7 Jul 2004 10:42:55 +0000 (GMT) Received: from barry.mail.mindspring.net (barry.mail.mindspring.net [207.69.200.25]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0E3FE43D53 for ; Wed, 7 Jul 2004 10:42:55 +0000 (GMT) (envelope-from tlambert2@mindspring.com) Received: from [192.168.167.39] (helo=wamui01.slb.atl.earthlink.net) by barry.mail.mindspring.net with esmtp (Exim 3.33 #1) id 1Bi9t6-0002GV-00; Wed, 07 Jul 2004 06:42:52 -0400 Message-ID: <7040592.1089196972901.JavaMail.root@wamui01.slb.atl.earthlink.net> Date: Wed, 7 Jul 2004 03:42:52 -0700 (GMT-07:00) From: Terry Lambert To: Petr Holub , threads@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Mailer: Earthlink Zoo Mail 1.0 Subject: Re: sem_wait as cancellation point X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Terry Lambert List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2004 10:42:55 -0000 Petr Holub wrote: > I wonder why sem_wait is not a pthread cancellation point on > FreeBSD-5. According to Linux man page: The Linux man page is totally irrelevant. Here's what the standards say: http://www.opengroup.org/onlinepubs/009695399/functions/xsh_chap02_09.html " Cancellation Points Cancellation points shall occur when a thread is executing the following functions: accept() mq_timedsend() putpmsg() sigtimedwait() ... getmsg() pselect() sem_wait() waitpid() ..." So it looks like it needs to be a cancellation point, if the thread cancellation option is fully supported. So before anyone else says it... "patches welcome". -- Terry From owner-freebsd-threads@FreeBSD.ORG Wed Jul 7 12:18:38 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 459EB16A4CE for ; Wed, 7 Jul 2004 12:18:38 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id A00E743D3F for ; Wed, 7 Jul 2004 12:18:37 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i67CIafP012677; Wed, 7 Jul 2004 08:18:36 -0400 (EDT) Date: Wed, 7 Jul 2004 08:18:36 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Petr Holub In-Reply-To: <000d01c46400$732b6d60$2603fb93@KLOBOUCEK> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: Re: sem_wait as cancellation point X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2004 12:18:38 -0000 On Wed, 7 Jul 2004, Petr Holub wrote: > Hi all, > > I wonder why sem_wait is not a pthread cancellation point on > FreeBSD-5. According to Linux man page: sem_wait() is a cancellation point in libpthread. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jul 8 05:15:04 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6685216A4CE for ; Thu, 8 Jul 2004 05:15:04 +0000 (GMT) Received: from anor.ics.muni.cz (anor.ics.muni.cz [147.251.4.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id BC9B643D31 for ; Thu, 8 Jul 2004 05:15:03 +0000 (GMT) (envelope-from hopet@ics.muni.cz) Received: from KLOBOUCEK (w54-183.fi.muni.cz [147.251.54.183]) (user=hopet@META mech=LOGIN bits=0) by anor.ics.muni.cz (8.12.1/8.12.1) with ESMTP id i685F1nD031555 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NO); Thu, 8 Jul 2004 07:15:02 +0200 From: "Petr Holub" To: "Daniel Eischen" Date: Thu, 8 Jul 2004 07:14:59 +0200 Message-ID: <007d01c464aa$840deea0$b736fb93@KLOBOUCEK> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook 8.5, Build 4.71.2377.0 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 In-Reply-To: Importance: Normal X-Muni-Spam-TestIP: 147.251.54.183 X-Muni-Virus-Test: Clean cc: threads@freebsd.org Subject: RE: sem_wait as cancellation point X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jul 2004 05:15:04 -0000 > sem_wait() is a cancellation point in libpthread. No, it is not - at least according to my experience of 5.2.1-RELEASE-p9. When I use following snipplet of code, it keeps hanging in the pthread_join: ---------------------------------------------------------------- /* Example to demonstate non-cancellable behavior of sem_wait in phtread * environment. I've tested it on FreeBSD 5.2.1-RELEASE-p9 and the program * was compiled using: * cc -D_THREAD_SAFE -o sem_wait sem_wait.c -pthread -lc_r */ #include #include #include #include pthread_t test_thread; sem_t my_semaphore; void * testsem (void *data) { printf("Initing semaphore.\n"); sem_init(&my_semaphore, 0, 0); printf("Posting semaphore.\n"); sem_post(&my_semaphore); printf("Waiting for semaphore (this should pass).\n"); sem_wait(&my_semaphore); printf("Waiting for semaphore (this should hang).\n"); sem_wait(&my_semaphore); printf("Got over sem_wait - strange.\n"); } int main() { void *thread_result; printf("Creating thread.\n"); pthread_create(&test_thread, NULL, testsem, (void *) NULL); printf("Sleeping for 5 secs.\n"); sleep(3); printf("Trying to cancel the thread.\n"); pthread_cancel(test_thread); sleep(1); printf("Trying to join the thread.\n"); pthread_join(test_thread, &thread_result); printf("Finished.\n"); return(EXIT_SUCCESS); } ---------------------------------------------------------------- Am I doing something wrong? Thanks, Petr ================================================================ Petr Holub CESNET z.s.p.o. Supercomputing Center Brno Zikova 4 Institute of Compt. Science 162 00 Praha 6, CZ Masaryk University Czech Republic Botanicka 68a, 60200 Brno, CZ e-mail: Petr.Holub@cesnet.cz phone: +420-549493944 fax: +420-541212747 e-mail: hopet@ics.muni.cz From owner-freebsd-threads@FreeBSD.ORG Thu Jul 8 10:24:57 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 078E516A4CE for ; Thu, 8 Jul 2004 10:24:57 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 888EF43D39 for ; Thu, 8 Jul 2004 10:24:56 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i68AOtfP029990; Thu, 8 Jul 2004 06:24:55 -0400 (EDT) Date: Thu, 8 Jul 2004 06:24:55 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Petr Holub In-Reply-To: <007d01c464aa$840deea0$b736fb93@KLOBOUCEK> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: RE: sem_wait as cancellation point X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jul 2004 10:24:57 -0000 On Thu, 8 Jul 2004, Petr Holub wrote: > > sem_wait() is a cancellation point in libpthread. > > No, it is not - at least according to my experience of 5.2.1-RELEASE-p9. > When I use following snipplet of code, it keeps hanging in the pthread_join: You're using libc_r. libpthread is in -current. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Thu Jul 8 10:27:49 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D0B4016A4CE for ; Thu, 8 Jul 2004 10:27:49 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 743DB43D53 for ; Thu, 8 Jul 2004 10:27:49 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i68ARmfP000471; Thu, 8 Jul 2004 06:27:48 -0400 (EDT) Date: Thu, 8 Jul 2004 06:27:48 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Petr Holub In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: threads@freebsd.org Subject: RE: sem_wait as cancellation point X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Jul 2004 10:27:49 -0000 On Thu, 8 Jul 2004, Daniel Eischen wrote: > On Thu, 8 Jul 2004, Petr Holub wrote: > > > > sem_wait() is a cancellation point in libpthread. > > > > No, it is not - at least according to my experience of 5.2.1-RELEASE-p9. > > When I use following snipplet of code, it keeps hanging in the pthread_join: > > You're using libc_r. libpthread is in -current. -bash-2.05b$ uname -a FreeBSD sirius 5.2-CURRENT FreeBSD 5.2-CURRENT #17: Sat Jul 3 20:04:16 EDT 2004 root@sirius:/usr/obj/usr/src/sys/sirius i386 -bash-2.05b$ ./sem_wait Creating thread. Sleeping for 5 secs. Initing semaphore. Posting semaphore. Waiting for semaphore (this should pass). Waiting for semaphore (this should hang). Trying to cancel the thread. Trying to join the thread. Finished. -- Dan Eischen From owner-freebsd-threads@FreeBSD.ORG Fri Jul 9 03:50:58 2004 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1935E16A4CE for ; Fri, 9 Jul 2004 03:50:58 +0000 (GMT) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5103643D45 for ; Fri, 9 Jul 2004 03:50:57 +0000 (GMT) (envelope-from eischen@vigrid.com) Received: from mail.pcnet.com (mail.pcnet.com [204.213.232.4]) by mail.pcnet.com (8.12.10/8.12.1) with ESMTP id i693n6fP028381; Thu, 8 Jul 2004 23:49:06 -0400 (EDT) Date: Thu, 8 Jul 2004 23:49:06 -0400 (EDT) From: Daniel Eischen X-Sender: eischen@pcnet5.pcnet.com To: Andrew Gallatin In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Julian Elischer cc: freebsd-threads@freebsd.org Subject: Re: pthread switch (was Odd KSE panic) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: deischen@freebsd.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jul 2004 03:50:58 -0000 On Tue, 6 Jul 2004, Daniel Eischen wrote: > On Tue, 6 Jul 2004, Andrew Gallatin wrote: > > > > > > does your worker thread loop to check if there is more work before > > > waiting to be notified? > > > > Yes. He takes a mutex, loops over all completed events, sending > > pthread_signals as required, then releases the mutex and sleeps via > > an ioctl. > > Note that he is holding the mutex while calling pthread_cond_signal(). > If we enable preemption in pthread_cond_signal(), then I suspect it > would be even worse than without preemption. > > I think the only place where it is sane to enable preemption is > on pthread_mutex_unlock(). Wewll, I just took a look at this. I had already added a preemption point in pthread_mutex_unlock(): $ cvs log -r1.38 lib/libpthread/thread/thr_mutex.c ... description: ---------------------------- revision 1.38 date: 2003/07/18 02:46:29; author: deischen; state: Exp; lines: +10 -6 Add a preemption point when a mutex or condition variable is handed-off/signaled to a higher priority thread. Note that when there are idle KSEs that could run the higher priority thread, we still add the preemption point because it seems to take the kernel a while to schedule an idle KSE. The drawbacks are that threads will be swapped more often between CPUs (KSEs) and that there will be an extra userland context switch (the idle KSE is still woken and will probably resume the preempted thread). We'll revisit this if and when idle CPU/KSE wakeup times improve. Note that the priority of a runnable thread must be strictly greater than the currently running thread in order for preemption to occur. -- Dan Eischen