From owner-freebsd-threads@FreeBSD.ORG Sun May 20 23:11:33 2007 Return-Path: X-Original-To: threads@freebsd.org Delivered-To: freebsd-threads@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BE7CE16A421; Sun, 20 May 2007 23:11:33 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 70C3613C43E; Sun, 20 May 2007 23:11:33 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.101] (c-71-231-138-78.hsd1.or.comcast.net [71.231.138.78]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id l4KNBVIf003485 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Sun, 20 May 2007 19:11:32 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Sun, 20 May 2007 16:11:29 -0700 (PDT) From: Jeff Roberson X-X-Sender: jroberson@10.0.0.1 To: smp@freebsd.org, threads@freebsd.org, performance@freebsd.org Message-ID: <20070520161051.L632@10.0.0.1> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: sched_lock && thread_lock() (fwd) X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 May 2007 23:11:33 -0000 In case any of you missed it, I sent this mail to arch@. Please keep the discussion there. Thanks, Jeff ---------- Forwarded message ---------- Date: Sun, 20 May 2007 16:07:53 -0700 (PDT) From: Jeff Roberson To: arch@freebsd.org Subject: sched_lock && thread_lock() Attilio and I have been working on addressing the increasing problem of sched_lock contention on -CURRENT. Attilio has been addressing the parts of the kernel which do not need to fall under the scheduler lock and moving them into seperate locks. For example, the ldt/gdt lock and clock lock which were committed earlier. Also, using atomics for the vmcnt structure. I have been working on an approach to using thread locks rather than a global scheduler lock. The design is similar to Solaris's container locks, but the details are different. The basic idea is to have a pointer in the thread structure that points at a spinlock that protects the thread. This spinlock may be one of the scheduler lock, a turnstile lock, or a sleep queue lock. As the thread changes state from running to blocked on a lock or sleeping the lock changes with it. This has several advantages. The majority of the kernel simply calls thread_lock() which figures out the details. The kernel then knows nothing of the particulars of the scheduler locks, and the schedulers are free to implement them in any way that they like. Furthermore, in some cases the locking is reduced, because locking the thread has the side effect of locking the container. This patch does not implement per-cpu scheduler locks. It just changes the kernel to support this model. I have a fork of ULE in development that runs with per-cpu locks, but it is not ready yet. This means that there should be very little change in system performance until the scheduler catches up. In fact, on a 2cpu system the difference is immeasurable or almost so on every workload I have tested. On an 8way opteron system the results vary between +10% on some reasonable workloads and -15% on super-smack, which has some inherent problems that I believe are not exposing real performance problems with this patch. This has also been tested extensively by Kris and myself on a variety of machines and I believe it to be fairly solid. The only thing remaining to do is fix rusage so that it does not rely on a global scheduler lock. I am posting the patch here in case anyone with specific knowledge of turnstiles, sleepqueues, or signals would like to review it, and as a general heads up to people interested in where the kernel is headed. This will apply to current just prior to my kern_clock.c commits. I will re-merge and update again in the next few days, probably after we sort out rusage. http://people.freebsd.org/~jeff/threadlock.diff Thanks, Jeff _______________________________________________ freebsd-arch@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-arch To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-threads@FreeBSD.ORG Mon May 21 11:08:47 2007 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B225616A480 for ; Mon, 21 May 2007 11:08:47 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [69.147.83.40]) by mx1.freebsd.org (Postfix) with ESMTP id A0F6913C458 for ; Mon, 21 May 2007 11:08:47 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id l4LB8lYI028930 for ; Mon, 21 May 2007 11:08:47 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id l4LB8kQp028926 for freebsd-threads@FreeBSD.org; Mon, 21 May 2007 11:08:46 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 21 May 2007 11:08:46 GMT Message-Id: <200705211108.l4LB8kQp028926@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: linimon set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 11:08:47 -0000 Current FreeBSD problem reports Critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- s threa/76690 threads fork hang in child for -lc_r 1 problem total. Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/20016 threads pthreads: Cannot set scheduling timer/Cannot set virtu s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s bin/32295 threads pthread dont dequeue signals s threa/34536 threads accept() blocks other threads o kern/38549 threads the procces compiled whith pthread stopped in pthread_ s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/49087 threads Signals lost in programs linked with libc_r s kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOC o threa/70975 threads unexpected and unreliable behaviour when using SYSV se o threa/72429 threads threads blocked in stdio (fgets, etc) are not cancella o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag s threa/76694 threads fork cause hang in dup()/close() function in child (-l o threa/79683 threads svctcp_create() fails if multiple threads call at the o threa/80435 threads panic on high loads o threa/83914 threads [libc] popen() doesn't work in static threaded program s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/85160 threads [libthr] [patch] libobjc + libpthread/libthr crash pro f threa/90278 threads libthr, ULE and -current produces >100% WCPU with apac o kern/91266 threads [threads] Trying sleep, but thread marked as sleeping s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc f threa/98256 threads gnome-system-monitor core dumps from pthread_testcance s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r o threa/101323 threads fork(2) in threaded programs broken. o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/110636 threads gdb(1): using gdb with multi thread application with l 29 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- s kern/19247 threads uthread_sigaction.c does not do anything wrt SA_NOCLDW s kern/22190 threads A threaded read(2) from a socketpair(2) fd can sometim s threa/30464 threads pthread mutex attributes -- pshared s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/69020 threads pthreads library leaks _gc_mutex o threa/79887 threads [patch] freopen() isn't thread-safe o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/81534 threads [libc_r] [patch] libc_r close() will fail on any fd ty o threa/110306 threads apache 2.0 segmentation violation when calling gethost 10 problems total.