From owner-freebsd-current@FreeBSD.ORG Thu Jun 17 19:35:38 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DE69716A4CE; Thu, 17 Jun 2004 19:35:38 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5DE1543D4C; Thu, 17 Jun 2004 19:35:38 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5HJJ8SZ046360; Thu, 17 Jun 2004 15:19:08 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5HJJ72I046357; Thu, 17 Jun 2004 15:19:07 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Thu, 17 Jun 2004 15:19:07 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Jon Noack In-Reply-To: <40D1D3D1.9010209@alumni.rice.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: mike cc: Julian Elischer cc: freebsd-threads@freebsd.org cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 19:35:39 -0000 On Thu, 17 Jun 2004, Jon Noack wrote: > > ULE seems to do a very good job of scheduling interactive tasks over > > other workloads, resulting in a very "snappy" feel on my boxes, > > despite heavy CPU load from background builds, etc. The workload I > > looked at had no real "interactive" component, although it was a > > latency-centric RPC test, so timely hand-off as well as high > > throughput would be important. I know that Jeff's measurement work > > on ULE had a substantial focus on deadlines -- whether or not ULE was > > timely in scheduling tasks, etc, and that he demonstrated that it was > > much stronger than most other available schedulers in this area. > > Well, I switched to 4BSD and noticed this right away. I occasionally > get sound hiccups under heavy i/o with ULE [1], but with 4BSD it's a bit > ridiculous. There's a PR open about it (can't remember which off the > top of my head), but sound will skip and slow down (play at half speed > or slower). I know the sound locking is not all it should be, so this > may not be a representative test. Regardless, my workstations are back > on ULE... Hrm. So, I'm not really a scheduler guy, but I have some ideas about how to investigate what's going on. If you feel like playing with kernel tracing facilities, there are some really neat things you can do with ktr(4). It allows you took hook context switch and interrupt delivery events and dump a trace to userspace. One possible cause of the symptoms you're seeing is poor interrupt handling response time -- possibly a failure to schedule the ithread in a timely manner. It would be quite interesting to compare the latency in scheduling (and running) the ithread across ULE and 4BSD and see how well each is managing to do in getting it running quickly. I'm not sure there's a tutorial on how to get KTR(4) running -- I found the man page helpful, but a little confusing. What you'll want to do, assuming you're willing to get into this, is turn on KTR tracing with KTR_INTR and KTR_PROC. You'll need to set the flags in the kernel compile, and then set them using a sysctl. You can use ktrdump to dump a record trace -- I'd use a number of flags present to add timestamps, cpu numbers if SMP, etc. With some post-processing, it should be possible to generate a distribution of "interrupt time" to "running interrupt handler". It would be useful to demonstrate to oneself that ithreads are running quickly, preempting things they need to, etc. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research