From owner-freebsd-performance@FreeBSD.ORG Sun Sep 16 02:50:54 2007 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A140C16A417; Sun, 16 Sep 2007 02:50:54 +0000 (UTC) (envelope-from kurt@intricatesoftware.com) Received: from mail1.intricatesoftware.com (cl-18.ewr-01.us.sixxs.net [IPv6:2001:4830:1200:11::2]) by mx1.freebsd.org (Postfix) with ESMTP id 428A813C45A; Sun, 16 Sep 2007 02:50:54 +0000 (UTC) (envelope-from kurt@intricatesoftware.com) Received: from seraph.intricatesoftware.com (relay@localhost.intricatesoftware.com [IPv6:::1]) by mail1.intricatesoftware.com (8.14.1/8.13.4) with ESMTP id l8G2ooGX020527; Sat, 15 Sep 2007 22:50:51 -0400 (EDT) Received: from seraph.intricatesoftware.com (truk@localhost.intricatesoftware.com [127.0.0.1]) by seraph.intricatesoftware.com (8.14.1/8.14.1) with ESMTP id l8G2op40022047; Sat, 15 Sep 2007 22:50:51 -0400 (EDT) Received: (from truk@localhost) by seraph.intricatesoftware.com (8.14.1/8.14.1/Submit) id l8G2opq6018558; Sat, 15 Sep 2007 22:50:51 -0400 (EDT) X-Authentication-Warning: seraph.intricatesoftware.com: truk set sender to kurt@intricatesoftware.com using -f From: Kurt Miller To: freebsd-java@freebsd.org Date: Sat, 15 Sep 2007 22:50:50 -0400 User-Agent: KMail/1.9.7 References: <46EC1B55.4060202@FreeBSD.org> In-Reply-To: <46EC1B55.4060202@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200709152250.50879.kurt@intricatesoftware.com> X-SMTP-Vilter-Version: 1.3.6 X-SMTP-Vilter-Virus-Backend: clamd X-SMTP-Vilter-Status: clean X-SMTP-Vilter-clamd-Virus-Status: clean X-Spamd-Symbols: ALL_TRUSTED,AWL X-SMTP-Vilter-Spam-Backend: spamd X-Spam-Score: -1.3 X-Spam-Threshold: 5.0 X-Spam-Probability: -0.3 X-Its-A-Nuisance: This is spam X-SMTP-Vilter-Unwanted-Backend: attachment X-SMTP-Vilter-attachment-Unwanted-Status: clean X-Mailman-Approved-At: Sun, 16 Sep 2007 16:07:04 +0000 Cc: java@freebsd.org, Kris Kennaway , performance@freebsd.org Subject: Re: Massive performance loss from OS::sleep hack X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Sep 2007 02:50:54 -0000 On Saturday 15 September 2007 01:50:13 pm Kris Kennaway wrote: > Hi, > > I have been running the volano java benchmark > (http://www.volano.com/benchmarks.html) on an 8-core i386 system, and > out of the box jdk15 on FreeBSD performs extremely poorly. The system > is more than 90% idle, and profiling shows that the ~800 threads in the > benchmark are spending most of their time doing short nanosleep() calls. > > > I traced it to the following FreeBSD-specific hack in the jdk: > > // XXXBSD: understand meaning and workaround related to yield > ... > // XXXBSD: done differently in 1.3.1, take a look > int os::sleep(Thread* thread, jlong millis, bool interruptible) { > assert(thread == Thread::current(), "thread consistency check"); > ... > > if (millis <= 0) { > // NOTE: workaround for bug 4338139 > if (thread->is_Java_thread()) { > ThreadBlockInVM tbivm((JavaThread*) thread); > // BSDXXX: Only use pthread_yield here and below if the system thread > // scheduler gives time slices to lower priority threads when yielding. > #ifdef __FreeBSD__ > os_sleep(MinSleepInterval, interruptible); > #else > pthread_yield(); > #endif > > When I removed this hack (i.e. revert to pthread_yield()) I got an > immediate 7-fold performance increase, which brings FreeBSD performance > on par with Solaris. > > What is the reason why this code is necessary? Does FreeBSD's > sched_yield() really have different semantics to the other operating > systems, or was this a libkse bug that was being worked around? Hello Kris, I recall why I added the os_sleep() call. While working on the certification testing one of the JCK tests was deadlocking. The test itself was faulty in that it created a high priority thread in a tight yield loop. Since pthread_yield() on a single processor system will not yield to lower priority threads, the higher priority thread effectively blocked the lower priority thread from running and the test deadlocked. I filed a formal appeal to have the JCK test corrected. However since the api states: http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html#yield() "Causes the currently executing thread object to temporarily pause and allow other threads to execute." the appeal was denied. I further argued that many publications written by or or-authored by Sun employees state that Thread.yield() can not be relied upon in this fashion: "However, I believe there are other less authoritative sources of information that support the position that yield() can not expected to allow lower priority threads to execute in all cases. For example the following Sun document describes yield() as advisory only and not to depend on it for correctness: http://java.sun.com/j2se/1.5.0/docs/guide/vm/thread-priorities.html#general The document also refers to a Sun published book, "Effective Java Programming Language Guide" that describes yield() should not be relied on for correctness and portability. Another book I have specifically addresses the topic. It states that most schedulers do not stop the yielding thread from running in favor of a thread of lower priority. This is from the Sybex "Complete Java 2 Certification Study Guide" co-authored by a Sun employee. The publications I have referred to above present a case that the api description of yield() doesn't fully describe the expected behavior of the function when combined with thread priorities. While they are not the authoritive publications on the function, I think they represent the general historical interpretation of the expected behavior." In the end I was not able to convince Sun to change the JCK test so the os_sleep() call was added. I see in my notes that kse on a single cpu system had the issue but thr didn't (this is on 6.1-release). Perhaps now that thr is the default on 7.0 this hack can be made conditional and only applied to < 7.0. The following are programs I wrote when I isolated the problem. If the c program runs ok on 7.0 for both single cpu and mp then remove the os_sleep() and try the java program. If that works too then you're clear to make the os_sleep() hack only apply to < 7.0 and still be able to pass the certification tests. Regards, -Kurt ----------- #include #include #include #include #include #include #include #include volatile int init=0; volatile int interrupt=0; static void * yielder(void *arg) { init = 1; while (1) { pthread_yield(); } } static void sighandler(int sig) { interrupt = 1; printf("sighandler\n"); static void sighandler(int sig) { interrupt = 1; printf("sighandler\n"); } static void waitForInit() { struct timespec t, rt; static void waitForInit() { struct timespec t, rt; while (init == 0) { t.tv_sec = 0; t.tv_nsec = 100000; nanosleep(&t, &rt); } } static void waitForInterrupt() { struct timespec t, rt; while (interrupt == 0) { t.tv_sec = 0; t.tv_nsec = 100000; nanosleep(&t, &rt); } } int main(int argc, char *argv[]) { pthread_t yldr; pthread_attr_t attr; struct sigaction act; /* Install a signal handler for SIGUSR1 */ sigemptyset (&act.sa_mask); sigaddset (&act.sa_mask, SIGUSR1); act.sa_handler = sighandler; act.sa_flags = 0; sigaction (SIGUSR1, &act, NULL); pthread_attr_init(&attr); pthread_attr_setschedpolicy(&attr, SCHED_FIFO); pthread_create(&yldr, &attr, yielder, NULL); pthread_setprio(yldr, 16); waitForInit(); if(pthread_kill(yldr, SIGUSR1) != 0) printf("pthread_kill failed with errno = %d\n", errno); waitForInterrupt(); } --------------------- public class yieldTest { public static void main(String[] args) { yielderThread thread = new yielderThread(); thread.setPriority(Thread.NORM_PRIORITY+1); thread.start(); thread.waitForInit(); thread.interrupt(); try { thread.join(); } catch (java.lang.InterruptedException ie) { } } } class yielderThread extends Thread { volatile boolean init = false; void waitForInit() { while (!init) { try { Thread.sleep(100); } catch (InterruptedException e) { } } } public void run() { init = true; while(!isInterrupted()) { yield(); } } }