From owner-freebsd-current@FreeBSD.ORG Wed Jun 16 05:25:52 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3562516A4CE; Wed, 16 Jun 2004 05:25:52 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9093543D41; Wed, 16 Jun 2004 05:25:51 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.11/8.12.11) with ESMTP id i5G5NtMq083977; Wed, 16 Jun 2004 01:23:55 -0400 (EDT) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i5G5NteE083974; Wed, 16 Jun 2004 01:23:55 -0400 (EDT) (envelope-from robert@fledge.watson.org) Date: Wed, 16 Jun 2004 01:23:55 -0400 (EDT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Julian Elischer In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-threads@freebsd.org cc: mike cc: freebsd-current@freebsd.org Subject: Re: Possible Threading problem with -CURRENT / MySQL? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jun 2004 05:25:52 -0000 On Tue, 15 Jun 2004, Julian Elischer wrote: > On Mon, 14 Jun 2004, mike wrote: > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x > > as well. it sounds like scheduler/threading is to blame but we were not > > able to give sufficient or proper motivation to the folks who could > > examine this deeper - we even offered $500 cash to whomever stepped up to > > help resolve this. > > > > linux runs almost 2x as fast on the same hardware with no configuring - > > and we get nearly the same results running in single CPU mode vs. dual CPU > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x > > combination. > > You complained about this some time ago and you have still not responded > with the information I suggest.. I sent this to Jeremy privately, since it was just some preliminary measurements, but figured I'd send it publically since the results were interesting (if tentative, I need to do a lot more work to make them useful. There are a number of variables I need to look at including: - Disabling HTT. A chat with Scott Long this evening suggests that HTT may be substantially hurting the test cases given increased IPIs, etc. Unfortunately, it looks like I can't easily twiddle HTT without being local to the machine, and I'm at home right now (it being 1:30am and all). Removing HTT may help substantially with the dip in performance in the SMP configuration. - I'd like to compare against RELENG_4 and a recent Linux kernel. Unfortunately, the box is configured for neither right now. - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd like to try SCHED_4BSD. - This was without adaptive mutexes, which seem to be helpful for others, so I should give them a try. I don't have any amd64 hardware, so I don't know what if any role it will play in the results. The performance drop observed in the report appears to be on amd64 (I may have misread). Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research ---- Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT) From: Robert Watson To: JG Subject: Re: Possible Threading problem with -CURRENT / MySQL? On Tue, 15 Jun 2004, JG wrote: > Fwiw, it has to be something that was committed between May 18th and > yesterday. ~May 18th was the last time I built -CURRENT during my last > round of testing and I did not have any of these problems. Then someone > emailed me recently and said there were some commits that might effect > the outcome of the mysql benchmarks. Ok, so these results are on a dual-processor XEON + hyperthreads, so four logical processors. I used two dates off CVS, 20040515 and 20040615. I also benchmarked my netperf branch. I don't have RELENG_4 on the box, but might be able to load RELENG_4 on it later this week. In each case, I took ten samples, dropped the first value as getting into the cache, and took the mean of the rest. For this test, I used the select test; I'll try the other smack query set tomorrow. In each case, I ran with "10 1000" as the arguments to the test. I used the default threading configuration in -CURRENT, which is libpthread (libkse). Mean Stdev 20040515-UP 4752.27 14.63 20040515-SMP 2550.35 19.23 20040615-UP 4898.71 22.39 20040615-SMP 2666.93 32.01 Netperf-UP-giant 4902.41 14.3 Netperf-SMP-giant 2566.18 16.83 Netperf-UP-mpsafe 4799.35 22.04 Netperf-SMP-mpsafe 3022.51 18.06 Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages the SMP numbers a fair amount due to additional IPIs without benefit. However, the numbers basically suggest that on my hardware, the UP configuration is marginally faster than it was last month, and that if you throw in the netperf branch, the SMP case is a moderate amount faster. This suggests that either I'm just lucky, or that the performance loss might be specific to the amd64 version of FreeBSD. I'm going to run some more numbers tomorrow and try to post something more rigorous to the -threads list. I don't have RELENG_4 on the box or Linux on the box, but I may get a chance to later this week. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research