From owner-freebsd-stable@FreeBSD.ORG Mon Dec 12 16:31:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11AE010656E7; Mon, 12 Dec 2011 16:31:01 +0000 (UTC) (envelope-from pieter@degoeje.nl) Received: from mx.utwente.nl (mx2.utsp.utwente.nl [130.89.2.13]) by mx1.freebsd.org (Postfix) with ESMTP id 703058FC1E; Mon, 12 Dec 2011 16:31:00 +0000 (UTC) Received: from nox-laptop.student.utwente.nl (nox-laptop.student.utwente.nl [130.89.160.140]) by mx.utwente.nl (8.12.10/SuSE Linux 0.7) with ESMTP id pBCFr4ms020823; Mon, 12 Dec 2011 16:53:04 +0100 From: Pieter de Goeje To: freebsd-current@freebsd.org Date: Mon, 12 Dec 2011 16:53:04 +0100 User-Agent: KMail/1.9.10 References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201112121653.04522.pieter@degoeje.nl> X-UTwente-MailScanner-Information: Scanned by MailScanner. Contact icts.servicedesk@utwente.nl for more information. X-UTwente-MailScanner: Found to be clean X-UTwente-MailScanner-From: pieter@degoeje.nl X-Spam-Status: No Cc: "O. Hartmann" , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:31:01 -0000 On Monday 12 December 2011 14:47:57 O. Hartmann wrote: > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > > O. In my spare time I do some stuff which can be considered "HPC". If I recall correctly the most loud supporters of the notion that SCHED_BSD is faster than SCHED_ULE are using more threads than there are cores, causing CPU core contention and more importantly unevenly distributed runtimes among threads, resulting in suboptimal execution times for their programs. Since I've never actually seen that code in question it's hard to say whether or not this "unfair" distribution actually results in lower throughput or that it simply violates an assumption in the code that each thread takes about as long to finish its task. Although I haven't actually benchmarked the two schedulers directly, I have no reason to suspect SCHED_ULE of suboptimal performance because: 1) A program model where there are N threads on N cores which take work items from a shared queue until it is empty has almost perfect scaling on SCHED_ULE (I get 398% CPU usage on a quadcore) 2) The same program on Linux (dual boot) compiled with exactly the same compiler and flags runs slightly slower. I think this has to do with VM differences. What I'm trying to say is that until someone actually shows some code which has demonstrably lower performance on SCHED_ULE and this is not caused by IMHO improper timing dependencies between threads I'd say that there is no cause for concern here. I actually expect performance differences between the two schedulers to show in problems which cause a lot more contention on the CPU cores and use lots of locks internally so threads are frequently waiting on each other, for instance the MySQL benchmarks done a couple of years ago by Kris Kennaway. Aside from algorithmic limitations (SCHED_BSD doesn't really scale all that well), there will always exist some problems in which SCHED_BSD is faster because it by chance has a better execution order for these problems... The good thing is people have a choice :-). I'm looking forward to the results of your benchmark. -- Pieter de Goeje