From owner-freebsd-current@FreeBSD.ORG Thu Jun 17 01:40:50 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5778F16A4CE; Thu, 17 Jun 2004 01:40:50 +0000 (GMT) Received: from rwcrmhc11.comcast.net (rwcrmhc11.comcast.net [204.127.198.35]) by mx1.FreeBSD.org (Postfix) with ESMTP id 38A0343D58; Thu, 17 Jun 2004 01:40:50 +0000 (GMT) (envelope-from julian@elischer.org) Received: from interjet.elischer.org ([24.7.73.28]) by comcast.net (rwcrmhc11) with ESMTP id <2004061701395801300gfbgme>; Thu, 17 Jun 2004 01:39:58 +0000 Received: from localhost (localhost.elischer.org [127.0.0.1]) by InterJet.elischer.org (8.9.1a/8.9.1) with ESMTP id SAA68093; Wed, 16 Jun 2004 18:39:57 -0700 (PDT) Date: Wed, 16 Jun 2004 18:39:55 -0700 (PDT) From: Julian Elischer To: Robert Watson In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jun 2004 01:40:50 -0000 Robert, can you try see if HZ=5000 (up from 100) changes the performance? On Wed, 16 Jun 2004, Robert Watson wrote: > > On Tue, 15 Jun 2004, Julian Elischer wrote: > > > On Mon, 14 Jun 2004, mike wrote: > > > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > > > as well. it sounds like scheduler/threading is to blame but we were not > > > able to give sufficient or proper motivation to the folks who could > > > examine this deeper - we even offered $500 cash to whomever stepped up to > > > help resolve this. > > > > > > linux runs almost 2x as fast on the same hardware with no configuring - > > > and we get nearly the same results running in single CPU mode vs. dual CPU > > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > > > combination. > > > > You complained about this some time ago and you have still not responded > > with the information I suggest.. > > I sent this to Jeremy privately, since it was just some preliminary > measurements, but figured I'd send it publically since the results were > interesting (if tentative, I need to do a lot more work to make them > useful. There are a number of variables I need to look at including: > > - Disabling HTT. A chat with Scott Long this evening suggests that HTT > may be substantially hurting the test cases given increased IPIs, etc. > Unfortunately, it looks like I can't easily twiddle HTT without being > local to the machine, and I'm at home right now (it being 1:30am and > all). Removing HTT may help substantially with the dip in performance > in the SMP configuration. > > - I'd like to compare against RELENG_4 and a recent Linux kernel. > Unfortunately, the box is configured for neither right now. > > - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd > like to try SCHED_4BSD. > > - This was without adaptive mutexes, which seem to be helpful for others, > so I should give them a try. > > I don't have any amd64 hardware, so I don't know what if any role it will > play in the results. The performance drop observed in the report appears > to be on amd64 (I may have misread). > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > > ---- > Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT) > From: Robert Watson > To: JG > Subject: Re: Possible Threading problem with -CURRENT / MySQL? > > > On Tue, 15 Jun 2004, JG wrote: > > > Fwiw, it has to be something that was committed between May 18th and > > yesterday. ~May 18th was the last time I built -CURRENT during my last > > round of testing and I did not have any of these problems. Then someone > > emailed me recently and said there were some commits that might effect > > the outcome of the mysql benchmarks. > > Ok, so these results are on a dual-processor XEON + hyperthreads, so four > logical processors. I used two dates off CVS, 20040515 and 20040615. I > also benchmarked my netperf branch. I don't have RELENG_4 on the box, but > might be able to load RELENG_4 on it later this week. In each case, I > took ten samples, dropped the first value as getting into the cache, and > took the mean of the rest. For this test, I used the select test; I'll > try the other smack query set tomorrow. In each case, I ran with "10 > 1000" as the arguments to the test. I used the default threading > configuration in -CURRENT, which is libpthread (libkse). > > Mean Stdev > 20040515-UP 4752.27 14.63 > 20040515-SMP 2550.35 19.23 > > 20040615-UP 4898.71 22.39 > 20040615-SMP 2666.93 32.01 > > Netperf-UP-giant 4902.41 14.3 > Netperf-SMP-giant 2566.18 16.83 > Netperf-UP-mpsafe 4799.35 22.04 > Netperf-SMP-mpsafe 3022.51 18.06 > > Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages > the SMP numbers a fair amount due to additional IPIs without benefit. > However, the numbers basically suggest that on my hardware, the UP > configuration is marginally faster than it was last month, and that if you > throw in the netperf branch, the SMP case is a moderate amount faster. > This suggests that either I'm just lucky, or that the performance loss > might be specific to the amd64 version of FreeBSD. I'm going to run some > more numbers tomorrow and try to post something more rigorous to the > -threads list. > > I don't have RELENG_4 on the box or Linux on the box, but I may get a > chance to later this week. > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Senior Research Scientist, McAfee Research > > >