From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 14:09:32 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 344E0106564A; Mon, 12 Dec 2011 14:09:32 +0000 (UTC) (envelope-from ohartman@mail.zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id E0AC78FC08; Mon, 12 Dec 2011 14:09:31 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1Ra6Ed-00011K-IT>; Mon, 12 Dec 2011 14:48:03 +0100 Received: from telesto.geoinf.fu-berlin.de ([130.133.86.198]) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1Ra6Ed-0004V9-FJ>; Mon, 12 Dec 2011 14:48:03 +0100 Message-ID: <4EE6060D.5060201@mail.zedat.fu-berlin.de> Date: Mon, 12 Dec 2011 14:47:57 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Current FreeBSD , freebsd-stable@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> In-Reply-To: <4EE22421.9060707@gmail.com> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig3C34C0DC945A3DBF809616AD" X-Originating-IP: 130.133.86.198 X-Mailman-Approved-At: Mon, 12 Dec 2011 14:14:46 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 14:09:32 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig3C34C0DC945A3DBF809616AD Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > Not fully right, boinc defaults to run on idprio 31 so this isn't an > issue. And yes, there are cases where SCHED_ULE shows much better > performance then SCHED_4BSD. [...] Do we have any proof at hand for such cases where SCHED_ULE performs much better than SCHED_4BSD? Whenever the subject comes up, it is mentioned, that SCHED_ULE has better performance on boxes with a ncpu > 2. But in the end I see here contradictionary statements. People complain about poor performance (especially in scientific environments), and other give contra not being the case. Within our department, we developed a highly scalable code for planetary science purposes on imagery. It utilizes present GPUs via OpenCL if present. Otherwise it grabs as many cores as it can. By the end of this year I'll get a new desktop box based on Intels new Sandy Bridge-E architecture with plenty of memory. If the colleague who developed the code is willing performing some benchmarks on the same hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most recent Suse. For FreeBSD I intent also to look for performance with both different schedulers available. O. --------------enig3C34C0DC945A3DBF809616AD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk7mBhMACgkQU6Ni+wtCKv9mzQD9EQm3oYTJl1UY826v0xiOzMeF nrbUBR3bAjsxSp3A2hYA/0TL0ltenxGrjE6h8DEiQg6ozCymbh7vkFCTBnHkZlaP =8zGJ -----END PGP SIGNATURE----- --------------enig3C34C0DC945A3DBF809616AD-- From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 15:13:03 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 043ED1065670; Mon, 12 Dec 2011 15:13:03 +0000 (UTC) (envelope-from vince@unsane.co.uk) Received: from unsane.co.uk (unsane-pt.tunnel.tserv5.lon1.ipv6.he.net [IPv6:2001:470:1f08:110::2]) by mx1.freebsd.org (Postfix) with ESMTP id 616528FC1B; Mon, 12 Dec 2011 15:13:02 +0000 (UTC) Received: from vhoffman-macbooklocal.local ([10.10.10.20]) (authenticated bits=0) by unsane.co.uk (8.14.4/8.14.4) with ESMTP id pBCFD0EV014133 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 12 Dec 2011 15:13:00 GMT (envelope-from vince@unsane.co.uk) Message-ID: <4EE619FC.4000601@unsane.co.uk> Date: Mon, 12 Dec 2011 15:13:00 +0000 From: Vincent Hoffman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> X-Enigmail-Version: 1.3.3 X-Enigmail-Draft-Status: 513 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 15:13:03 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/12/2011 13:47, O. Hartmann wrote: > >> Not fully right, boinc defaults to run on idprio 31 so this isn't an >> issue. And yes, there are cases where SCHED_ULE shows much better >> performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. It all a little old now but some if the stuff in http://people.freebsd.org/~kris/scaling/ covers improvements that were seen. http://jeffr-tech.livejournal.com/5705.html shows a little too, reading though Jeffs blog is worth it as it has some interesting stuff on SHED_ULE. I thought there were some more benchmarks floating round but cant find any with a quick google. Vince > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > > O. > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJO5hn7AAoJEF4mgOY1fXowOLAP/2EjhAFPb88NgKM0ieBb4X7R NSw/9HTiwcshkfEdvYjAzYZ0cUWetEuRfnPVnh+abwfJEmMzZkwA0KIz8UYGHHik 22Z2SWSVDiwZAluz0ca7Xc931ojbzrK/zVMbivqW3cvnz8P4oEnASiENnsoa89Jy Oskjd4QpAyIpB/AsYgc9FLT3kPX13fXC5bzw/zAPDsaupOYssRRlZu8nnqsEc1i1 IanLIPKLnIbpZTx75ehWxxRW8IjiQRvIe+7eBaDMhXO/Kvftotf0JzknrBnJezDQ ZdhiOTq7F1Pm3dxra+DNKD+Dw+xUCYPFq/kuyqrZNz44H3qwT60vDhvw0yDz6422 nNP11z2+G4M85sahBak5AmSHuyek7HWb6uIHHnfvwNKSX4ZsdS8MVBViNJjmCYtL PwuHDU3WdCes/vvKRNDopSp/s6RSLK9w3RT7jlMkaTu2Mmtw0BwGziDJ2pGaCQ14 68R5eO/SfNxoVp0g4lIzObyQR+//0OmALzElVK3VmHM9NoL3qZGCwBRLqjN5re82 dX6nsBr/DFJOpaFfdFLwPNyCNdNpg/WVegRkq2BEL/BaMISNiKzoVbM0Psh9gnb3 LW1j3LP2fOHhuN1bW3S31JmbNzvAnlRNynoNMldrwj5PWJY2HPk+mMFRjmRwdDTJ 9mhscz8++WRPvDZQXefl =XqaR -----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 15:52:00 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 102E61065670; Mon, 12 Dec 2011 15:52:00 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id C3CAA8FC12; Mon, 12 Dec 2011 15:51:59 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBCFpxf6073711; Mon, 12 Dec 2011 07:51:59 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBCFpxO9073710; Mon, 12 Dec 2011 07:51:59 -0800 (PST) (envelope-from sgk) Date: Mon, 12 Dec 2011 07:51:59 -0800 From: Steve Kargl To: "O. Hartmann" Message-ID: <20111212155159.GB73597@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Mon, 12 Dec 2011 16:13:38 +0000 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 15:52:00 -0000 On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > This comes up every 9 months or so, and must be approaching FAQ status. In a HPC environment, I recommend 4BSD. Depending on the workload, ULE can cause a severe increase in turn around time when doing already long computations. If you have an MPI application, simply launching greater than ncpu+1 jobs can show the problem. PS: search the list archives for "kargl and ULE". -- Steve From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:18:39 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54E8A1065673; Mon, 12 Dec 2011 16:18:39 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id AA9AB8FC18; Mon, 12 Dec 2011 16:18:38 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id C9CE3E6208; Mon, 12 Dec 2011 16:18:36 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=Q9z1bk1qmKZA XQ8nRStBeMc3/Jk=; b=SxXMh7kUdQYpi15Kt2J0Rs6fYC1olhC17WIs0zDjIT5D A1Iq96vlmLVpcO1A6qoh99oVO5W12+/fCMZtP1D+efVwW3aBMFT4sZTaoXUOlFDh rGWM1gOuR1TCFVUlHquestEdOYF2nPQGwZnksQNhdP74MsDVCh/kWz9Ns5moGS4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=RcWMeC 1y8njc525uJc8U1Tmzb2xH95iBdBPjvx9fu+qY8W+IegM0A5JKq62k7OQUsNg5df YhaIreE02yvjKlB4ycT5fpwQ1N+QZPLK56mfkiGjFav7JnbvhQh6Fy5NrFdv/ULi t+KDjFXsZ1d3YNfJatjra7CllK0LWd/TQoKCk= Received: from [192.168.1.120] (188-220-36-32.zone11.bethere.co.uk [188.220.36.32]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 50421E6200; Mon, 12 Dec 2011 16:18:36 +0000 (GMT) Message-ID: <4EE6295B.3020308@cran.org.uk> Date: Mon, 12 Dec 2011 16:18:35 +0000 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Steve Kargl References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> In-Reply-To: <20111212155159.GB73597@troutmask.apl.washington.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:18:39 -0000 On 12/12/2011 15:51, Steve Kargl wrote: > This comes up every 9 months or so, and must be approaching FAQ > status. In a HPC environment, I recommend 4BSD. Depending on the > workload, ULE can cause a severe increase in turn around time when > doing already long computations. If you have an MPI application, > simply launching greater than ncpu+1 jobs can show the problem. PS: > search the list archives for "kargl and ULE". This isn't something that can be fixed by tuning ULE? For example for desktop applications kern.sched.preempt_thresh should be set to 224 from its default. I'm wondering if the installer should ask people what the typical use will be, and tune the scheduler appropriately. -- Bruce Cran From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:31:01 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11AE010656E7; Mon, 12 Dec 2011 16:31:01 +0000 (UTC) (envelope-from pieter@degoeje.nl) Received: from mx.utwente.nl (mx2.utsp.utwente.nl [130.89.2.13]) by mx1.freebsd.org (Postfix) with ESMTP id 703058FC1E; Mon, 12 Dec 2011 16:31:00 +0000 (UTC) Received: from nox-laptop.student.utwente.nl (nox-laptop.student.utwente.nl [130.89.160.140]) by mx.utwente.nl (8.12.10/SuSE Linux 0.7) with ESMTP id pBCFr4ms020823; Mon, 12 Dec 2011 16:53:04 +0100 From: Pieter de Goeje To: freebsd-current@freebsd.org Date: Mon, 12 Dec 2011 16:53:04 +0100 User-Agent: KMail/1.9.10 References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201112121653.04522.pieter@degoeje.nl> X-UTwente-MailScanner-Information: Scanned by MailScanner. Contact icts.servicedesk@utwente.nl for more information. X-UTwente-MailScanner: Found to be clean X-UTwente-MailScanner-From: pieter@degoeje.nl X-Spam-Status: No Cc: "O. Hartmann" , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:31:01 -0000 On Monday 12 December 2011 14:47:57 O. Hartmann wrote: > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > > O. In my spare time I do some stuff which can be considered "HPC". If I recall correctly the most loud supporters of the notion that SCHED_BSD is faster than SCHED_ULE are using more threads than there are cores, causing CPU core contention and more importantly unevenly distributed runtimes among threads, resulting in suboptimal execution times for their programs. Since I've never actually seen that code in question it's hard to say whether or not this "unfair" distribution actually results in lower throughput or that it simply violates an assumption in the code that each thread takes about as long to finish its task. Although I haven't actually benchmarked the two schedulers directly, I have no reason to suspect SCHED_ULE of suboptimal performance because: 1) A program model where there are N threads on N cores which take work items from a shared queue until it is empty has almost perfect scaling on SCHED_ULE (I get 398% CPU usage on a quadcore) 2) The same program on Linux (dual boot) compiled with exactly the same compiler and flags runs slightly slower. I think this has to do with VM differences. What I'm trying to say is that until someone actually shows some code which has demonstrably lower performance on SCHED_ULE and this is not caused by IMHO improper timing dependencies between threads I'd say that there is no cause for concern here. I actually expect performance differences between the two schedulers to show in problems which cause a lot more contention on the CPU cores and use lots of locks internally so threads are frequently waiting on each other, for instance the MySQL benchmarks done a couple of years ago by Kris Kennaway. Aside from algorithmic limitations (SCHED_BSD doesn't really scale all that well), there will always exist some problems in which SCHED_BSD is faster because it by chance has a better execution order for these problems... The good thing is people have a choice :-). I'm looking forward to the results of your benchmark. -- Pieter de Goeje From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:01:07 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 07EF8106564A; Mon, 12 Dec 2011 16:01:07 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5EC3B8FC0C; Mon, 12 Dec 2011 16:01:05 +0000 (UTC) Received: by eekc50 with SMTP id c50so3330964eek.13 for ; Mon, 12 Dec 2011 08:01:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=SHPI/Eh4GHYyLbnl/U4ZuP6rptb+8Q9ldKyvRbK7Oi4=; b=mX7wboXam0RjDri+pOgLL53QpJqlOn0UsUBsVJViLlkWG+UPG7h4kq9Co+mHQgnVnO D3n9kN135RvCgJ2ZMMFBK9WR1GFN25fzHY9OPtojQrdAzMlgrBL7iVT8YiO1sqkLWzkC Vjy170H5PWDvbzTMx/skrjZ3zMDrTnpKVzh7U= Received: by 10.14.14.77 with SMTP id c53mr268271eec.85.1323703945006; Mon, 12 Dec 2011 07:32:25 -0800 (PST) Received: from ernst.jennejohn.org (p578E15DC.dip.t-dialin.net. [87.142.21.220]) by mx.google.com with ESMTPS id a60sm76476635eeb.4.2011.12.12.07.32.23 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 12 Dec 2011 07:32:24 -0800 (PST) Date: Mon, 12 Dec 2011 16:32:21 +0100 From: Gary Jennejohn To: Vincent Hoffman Message-ID: <20111212163221.33d0b8a2@ernst.jennejohn.org> In-Reply-To: <4EE619FC.4000601@unsane.co.uk> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 12 Dec 2011 16:31:31 +0000 Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:01:07 -0000 On Mon, 12 Dec 2011 15:13:00 +0000 Vincent Hoffman wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 12/12/2011 13:47, O. Hartmann wrote: > > > >> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >> issue. And yes, there are cases where SCHED_ULE shows much better > >> performance then SCHED_4BSD. [...] > > > > Do we have any proof at hand for such cases where SCHED_ULE performs > > much better than SCHED_4BSD? Whenever the subject comes up, it is > > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > > 2. But in the end I see here contradictionary statements. People > > complain about poor performance (especially in scientific environments), > > and other give contra not being the case. > It all a little old now but some if the stuff in > http://people.freebsd.org/~kris/scaling/ > covers improvements that were seen. > > http://jeffr-tech.livejournal.com/5705.html > shows a little too, reading though Jeffs blog is worth it as it has some > interesting stuff on SHED_ULE. > > I thought there were some more benchmarks floating round but cant find > any with a quick google. > > > Vince > > > > > Within our department, we developed a highly scalable code for planetary > > science purposes on imagery. It utilizes present GPUs via OpenCL if > > present. Otherwise it grabs as many cores as it can. > > By the end of this year I'll get a new desktop box based on Intels new > > Sandy Bridge-E architecture with plenty of memory. If the colleague who > > developed the code is willing performing some benchmarks on the same > > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > > recent Suse. For FreeBSD I intent also to look for performance with both > > different schedulers available. > > These observations are not scientific, but I have a CPU from AMD with 6 cores (AMD Phenom(tm) II X6 1090T Processor). My simple test was ``make buildkernel'' while watching the core usage with gkrellm. With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. I've never seen any value above 97% with gkrellm. With SCHED_ULE I never saw all 6 cores loaded this heavily. Usually 2 or more cores were at or below 90%. Not really that significant, but still a noticeable difference in apparent scheduling behavior. Whether the observed difference is due to some change in data from the kernel to gkrellm is beyond me. -- Gary Jennejohn From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:11:27 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8A0F106567B; Mon, 12 Dec 2011 16:11:27 +0000 (UTC) (envelope-from lars.engels@0x20.net) Received: from mail.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id 1E2838FC21; Mon, 12 Dec 2011 16:11:27 +0000 (UTC) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) by mail.0x20.net (Postfix) with ESMTP id 38ED66A661D; Mon, 12 Dec 2011 17:11:26 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.0x20.net Received: from mail.0x20.net ([217.69.76.211]) by mail.0x20.net (mail.0x20.net [217.69.76.211]) (amavisd-new, port 10024) with ESMTP id NOQH_WXjvpPe; Mon, 12 Dec 2011 17:11:26 +0100 (CET) Received: from [10.26.32.239] (ip-109-84-0-111.web.vodafone.de [109.84.0.111]) (Authenticated sender: lala) by mail.0x20.net (Postfix) with ESMTPA id DBC096A6619; Mon, 12 Dec 2011 17:11:22 +0100 (CET) References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <20111212163221.33d0b8a2@ernst.jennejohn.org> User-Agent: K-9 Mail for Android In-Reply-To: <20111212163221.33d0b8a2@ernst.jennejohn.org> MIME-Version: 1.0 From: Lars Engels Date: Mon, 12 Dec 2011 17:10:46 +0100 To: gljennjohn@googlemail.com,Vincent Hoffman Message-ID: <1a8a6d6f-6756-4cda-b4d6-b39d335678c1@email.android.com> X-Mailman-Approved-At: Mon, 12 Dec 2011 16:32:00 +0000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:11:27 -0000 Did you use -jX to build the world? _____________________________________________ Von: Gary Jennejohn Versendet am: Mon Dec 12 16:32:21 MEZ 2011 An: Vincent Hoffman CC: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Betreff: Re: SCHED_ULE should not be the default On Mon, 12 Dec 2011 15:13:00 +0000 Vincent Hoffman wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 12/12/2011 13:47, O. Hartmann wrote: > > > >> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >> issue. And yes, there are cases where SCHED_ULE shows much better > >> performance then SCHED_4BSD. [...] > > > > Do we have any proof at hand for such cases where SCHED_ULE performs > > much better than SCHED_4BSD? Whenever the subject comes up, it is > > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > > 2. But in the end I see here contradictionary statements. People > > complain about poor performance (especially in scientific environments), > > and other give contra not being the case. > It all a little old now but some if the stuff in > http://people.freebsd.org/~kris/scaling/ > covers improvements that were seen. > > http://jeffr-tech.livejournal.com/5705.html > shows a little too, reading though Jeffs blog is worth it as it has some > interesting stuff on SHED_ULE. > > I thought there were some more benchmarks floating round but cant find > any with a quick google. > > > Vince > > > > > Within our department, we developed a highly scalable code for planetary > > science purposes on imagery. It utilizes present GPUs via OpenCL if > > present. Otherwise it grabs as many cores as it can. > > By the end of this year I'll get a new desktop box based on Intels new > > Sandy Bridge-E architecture with plenty of memory. If the colleague who > > developed the code is willing performing some benchmarks on the same > > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > > recent Suse. For FreeBSD I intent also to look for performance with both > > different schedulers available. > > These observations are not scientific, but I have a CPU from AMD with 6 cores (AMD Phenom(tm) II X6 1090T Processor). My simple test was ``make buildkernel'' while watching the core usage with gkrellm. With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. I've never seen any value above 97% with gkrellm. With SCHED_ULE I never saw all 6 cores loaded this heavily. Usually 2 or more cores were at or below 90%. Not really that significant, but still a noticeable difference in apparent scheduling behavior. Whether the observed difference is due to some change in data from the kernel to gkrellm is beyond me. -- Gary Jennejohn _____________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:14:03 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75D9C106564A; Mon, 12 Dec 2011 16:14:03 +0000 (UTC) (envelope-from lars.engels@0x20.net) Received: from mail.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) by mx1.freebsd.org (Postfix) with ESMTP id EA3D88FC13; Mon, 12 Dec 2011 16:14:02 +0000 (UTC) Received: from mail.0x20.net (mail.0x20.net [217.69.76.211]) by mail.0x20.net (Postfix) with ESMTP id 439316A61CD; Mon, 12 Dec 2011 17:14:02 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.0x20.net Received: from mail.0x20.net ([217.69.76.211]) by mail.0x20.net (mail.0x20.net [217.69.76.211]) (amavisd-new, port 10024) with ESMTP id lhy24cgq8Rc2; Mon, 12 Dec 2011 17:14:02 +0100 (CET) Received: from [10.26.32.239] (ip-109-84-0-111.web.vodafone.de [109.84.0.111]) (Authenticated sender: lala) by mail.0x20.net (Postfix) with ESMTPA id 77B4F6A6016; Mon, 12 Dec 2011 17:13:59 +0100 (CET) References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> User-Agent: K-9 Mail for Android In-Reply-To: <20111212155159.GB73597@troutmask.apl.washington.edu> MIME-Version: 1.0 From: Lars Engels Date: Mon, 12 Dec 2011 17:13:08 +0100 To: Steve Kargl , "O. Hartmann" Message-ID: <158db475-0d35-4ff2-a101-cf056bdfcab8@email.android.com> X-Mailman-Approved-At: Mon, 12 Dec 2011 16:32:23 +0000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:14:03 -0000 Would it be possible to implement a mechanism that lets one change the scheduler on the fly? Afaik Solaris can do that. _____________________________________________ Von: Steve Kargl Versendet am: Mon Dec 12 16:51:59 MEZ 2011 An: "O. Hartmann" CC: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Betreff: Re: SCHED_ULE should not be the default On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. > This comes up every 9 months or so, and must be approaching FAQ status. In a HPC environment, I recommend 4BSD. Depending on the workload, ULE can cause a severe increase in turn around time when doing already long computations. If you have an MPI application, simply launching greater than ncpu+1 jobs can show the problem. PS: search the list archives for "kargl and ULE". -- Steve _____________________________________________ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:43:52 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F158106564A; Mon, 12 Dec 2011 16:43:52 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm1.ukr.net (fsm1.ukr.net [195.214.192.120]) by mx1.freebsd.org (Postfix) with ESMTP id D1BC88FC0A; Mon, 12 Dec 2011 16:43:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=jD/DGWevIEiPAWIQLQuAD22uufZYP5CNuRFCmUv51UA=; b=Dl0Gj5PbcN+1zz7mVWHE8/hPnfQxmsQdacvdqYFZvJtXvtn6Vuo9nFSEGw5UGmTAbc3eMexcRz+2Fi96kma36t1L9updJ005PSmNRxyWYNRFpgXcQd2MzgGq0oSNK7BFhPkdH7Bk1m/Yibe7yklOeunEW8URctf1hqGoINM0gRY=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm1.ukr.net with esmtpsa ID 1Ra8k5-0000sP-U4 ; Mon, 12 Dec 2011 18:28:41 +0200 Date: Mon, 12 Dec 2011 18:28:41 +0200 From: Ivan Klymenko To: Bruce Cran Message-ID: <20111212182841.31f669ca@nonamehost.> In-Reply-To: <4EE6295B.3020308@cran.org.uk> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Steve Kargl Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:43:52 -0000 =D0=92 Mon, 12 Dec 2011 16:18:35 +0000 Bruce Cran =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On 12/12/2011 15:51, Steve Kargl wrote: > > This comes up every 9 months or so, and must be approaching FAQ=20 > > status. In a HPC environment, I recommend 4BSD. Depending on the=20 > > workload, ULE can cause a severe increase in turn around time when=20 > > doing already long computations. If you have an MPI application,=20 > > simply launching greater than ncpu+1 jobs can show the problem. PS:=20 > > search the list archives for "kargl and ULE".=20 >=20 > This isn't something that can be fixed by tuning ULE? For example for=20 > desktop applications kern.sched.preempt_thresh should be set to 224 > from its default. I'm wondering if the installer should ask people > what the typical use will be, and tune the scheduler appropriately. >=20 This is by and large does not help in certain situations ... From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 19:16:07 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3CBC11065677; Mon, 12 Dec 2011 19:16:07 +0000 (UTC) (envelope-from lambert@lambertfam.org) Received: from sysmon.tcworks.net (sysmon.tcworks.net [65.66.76.4]) by mx1.freebsd.org (Postfix) with ESMTP id B16A08FC1E; Mon, 12 Dec 2011 19:16:06 +0000 (UTC) Received: from sysmon.tcworks.net (localhost [127.0.0.1]) by sysmon.tcworks.net (8.13.1/8.13.1) with ESMTP id pBCJ3Ua5079662; Mon, 12 Dec 2011 13:03:30 -0600 (CST) (envelope-from lambert@lambertfam.org) Received: (from lambert@localhost) by sysmon.tcworks.net (8.13.1/8.13.1/Submit) id pBCJ3UQl079661; Mon, 12 Dec 2011 13:03:30 -0600 (CST) (envelope-from lambert@lambertfam.org) X-Authentication-Warning: sysmon.tcworks.net: lambert set sender to lambert@lambertfam.org using -f Date: Mon, 12 Dec 2011 13:03:30 -0600 From: Scott Lambert To: Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Message-ID: <20111212190330.GA69380@sysmon.tcworks.net> Mail-Followup-To: Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111212170604.GA74044@troutmask.apl.washington.edu> User-Agent: Mutt/1.4.2.2i Cc: Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 19:16:07 -0000 On Mon, Dec 12, 2011 at 09:06:04AM -0800, Steve Kargl wrote: > Tuning kern.sched.preempt_thresh did not seem to help for > my workload. My code is a classic master-slave OpenMPI > application where the master runs on one node and all > cpu-bound slaves are sent to a second node. If I send > send ncpu+1 jobs to the 2nd node with ncpu's, then > ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The > last two jobs are assigned to the ncpu'th cpu, and > these ping-pong on the this cpu. AFAICT, it is a cpu > affinity issue, where ULE is trying to keep each job > associated with its initially assigned cpu. > > While one might suggest that starting ncpu+1 jobs > is not prudent, my example is just that. It is an > example showing that ULE has performance issues. > So, I now can start only ncpu jobs on each node > in the cluster and send emails to all other users > to not use those node, or use 4BSD and not worry > about loading issues. Does it meet your expectations if you start (j modulo ncpu) = 0 jobs on a node? -- Scott Lambert KC5MLE Unix SysAdmin lambert@lambertfam.org From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:30:39 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E510106564A; Mon, 12 Dec 2011 16:30:38 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2CB8A8FC12; Mon, 12 Dec 2011 16:30:37 +0000 (UTC) Received: by dakp5 with SMTP id p5so7144112dak.13 for ; Mon, 12 Dec 2011 08:30:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=+dryFrlmia5wNaD6PzjKbJ//sZuyakBBXiNiNL3Oq4Q=; b=lgfORXOm2V59Cfjmi+1ior8IoNtO4RvD68YAQJubbjcUkRg9uRh2blmihRprEjzK3F XXz0muY87H88yNPU+7SBGMsBVodsCdguO/hc8nhrugIpy77NclKf9Djthf3lsHR+fZeG F1lAK9kLQMEnkqQMVVhnGCN4P2Ag0+rg+6BRM= MIME-Version: 1.0 Received: by 10.68.211.5 with SMTP id my5mr36432945pbc.17.1323705877044; Mon, 12 Dec 2011 08:04:37 -0800 (PST) Sender: mdf356@gmail.com Received: by 10.68.197.198 with HTTP; Mon, 12 Dec 2011 08:04:37 -0800 (PST) In-Reply-To: <20111212163221.33d0b8a2@ernst.jennejohn.org> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <20111212163221.33d0b8a2@ernst.jennejohn.org> Date: Mon, 12 Dec 2011 08:04:37 -0800 X-Google-Sender-Auth: Z-9Mf-awLdKOtIu2Ho6RiSTBpfc Message-ID: From: mdf@FreeBSD.org To: gljennjohn@googlemail.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Mailman-Approved-At: Mon, 12 Dec 2011 22:57:12 +0000 Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Vincent Hoffman Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:30:39 -0000 On Mon, Dec 12, 2011 at 7:32 AM, Gary Jennejohn wrote: > On Mon, 12 Dec 2011 15:13:00 +0000 > Vincent Hoffman wrote: > >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On 12/12/2011 13:47, O. Hartmann wrote: >> > >> >> Not fully right, boinc defaults to run on idprio 31 so this isn't an >> >> issue. And yes, there are cases where SCHED_ULE shows much better >> >> performance then SCHED_4BSD. [...] >> > >> > Do we have any proof at hand for such cases where SCHED_ULE performs >> > much better than SCHED_4BSD? Whenever the subject comes up, it is >> > mentioned, that SCHED_ULE has better performance on boxes with a ncpu = > >> > 2. But in the end I see here contradictionary statements. People >> > complain about poor performance (especially in scientific environments= ), >> > and other give contra not being the case. >> It all a little old now but some if the stuff in >> http://people.freebsd.org/~kris/scaling/ >> covers improvements that were seen. >> >> http://jeffr-tech.livejournal.com/5705.html >> shows a little too, reading though Jeffs blog is worth it as it has some >> interesting stuff on SHED_ULE. >> >> I thought there were some more benchmarks floating round but cant find >> any with a quick google. >> >> >> Vince >> >> > >> > Within our department, we developed a highly scalable code for planeta= ry >> > science purposes on imagery. It utilizes present GPUs via OpenCL if >> > present. Otherwise it grabs as many cores as it can. >> > By the end of this year I'll get a new desktop box based on Intels new >> > Sandy Bridge-E architecture with plenty of memory. If the colleague wh= o >> > developed the code is willing performing some benchmarks on the same >> > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most >> > recent Suse. For FreeBSD I intent also to look for performance with bo= th >> > different schedulers available. >> > > > These observations are not scientific, but I have a CPU from AMD with > 6 cores (AMD Phenom(tm) II X6 1090T Processor). > > My simple test was ``make buildkernel'' while watching the core usage wit= h > gkrellm. > > With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. > I've never seen any value above 97% with gkrellm. > > With SCHED_ULE I never saw all 6 cores loaded this heavily. =A0Usually > 2 or more cores were at or below 90%. =A0Not really that significant, but > still a noticeable difference in apparent scheduling behavior. =A0Whether > the observed difference is due to some change in data from the kernel to > gkrellm is beyond me. SCHED_ULE is much sloppier about calculating which thread used a timeslice -- unless the timeslice went 100% to a thread, the fraction it used may get attributed elsewhere. So top's reporting of thread usage is not a useful metric. Total buildworld time is, potentially. Thanks, matthew From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:47:37 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5CC591065672; Mon, 12 Dec 2011 16:47:37 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 920F18FC17; Mon, 12 Dec 2011 16:47:36 +0000 (UTC) Received: by eekc50 with SMTP id c50so3382840eek.13 for ; Mon, 12 Dec 2011 08:47:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=Ne7LltZXZK7mD1SEiZFcIDyP8Mb797kkP51TT0gaTgc=; b=HpKS7+i6ORdEKlB0ZZOMNdQLfVhCj5U1R4RQ9U66CjKYvQBgsHLWhqZBxIOfpqu2Or NigZkoIhrXioa8tlSTgW2OEImo8bAE24IoqSjT7grHtNK0gDPGwVnRH7+1Dca0+C++UP /Ed0tVkrMrHxhwfOOE4LoQdxyW0c4ptsO0cm4= Received: by 10.14.4.134 with SMTP id 6mr2809056eej.183.1323708454160; Mon, 12 Dec 2011 08:47:34 -0800 (PST) Received: from ernst.jennejohn.org (p578E15DC.dip.t-dialin.net. [87.142.21.220]) by mx.google.com with ESMTPS id 53sm77232850eef.2.2011.12.12.08.47.31 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 12 Dec 2011 08:47:32 -0800 (PST) Date: Mon, 12 Dec 2011 17:47:30 +0100 From: Gary Jennejohn To: Lars Engels Message-ID: <20111212174730.0aee3e2c@ernst.jennejohn.org> In-Reply-To: <1a8a6d6f-6756-4cda-b4d6-b39d335678c1@email.android.com> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <20111212163221.33d0b8a2@ernst.jennejohn.org> <1a8a6d6f-6756-4cda-b4d6-b39d335678c1@email.android.com> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 12 Dec 2011 23:06:20 +0000 Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Vincent Hoffman Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:47:37 -0000 On Mon, 12 Dec 2011 17:10:46 +0100 Lars Engels wrote: > Did you use -jX to build the world? > I'm top posting since Lars did. It was buildkernel, not buildworld. Yes, -j6. > _____________________________________________ > Von: Gary Jennejohn > Versendet am: Mon Dec 12 16:32:21 MEZ 2011 > An: Vincent Hoffman > CC: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org > Betreff: Re: SCHED_ULE should not be the default > > > On Mon, 12 Dec 2011 15:13:00 +0000 > Vincent Hoffman wrote: > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > On 12/12/2011 13:47, O. Hartmann wrote: > > > > > >> Not fully right, boinc defaults to run on idprio 31 so this isn't an > > >> issue. And yes, there are cases where SCHED_ULE shows much better > > >> performance then SCHED_4BSD. [...] > > > > > > Do we have any proof at hand for such cases where SCHED_ULE performs > > > much better than SCHED_4BSD? Whenever the subject comes up, it is > > > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > > > 2. But in the end I see here contradictionary statements. People > > > complain about poor performance (especially in scientific environments), > > > and other give contra not being the case. > > It all a little old now but some if the stuff in > > http://people.freebsd.org/~kris/scaling/ > > covers improvements that were seen. > > > > http://jeffr-tech.livejournal.com/5705.html > > shows a little too, reading though Jeffs blog is worth it as it has some > > interesting stuff on SHED_ULE. > > > > I thought there were some more benchmarks floating round but cant find > > any with a quick google. > > > > > > Vince > > > > > > > > Within our department, we developed a highly scalable code for planetary > > > science purposes on imagery. It utilizes present GPUs via OpenCL if > > > present. Otherwise it grabs as many cores as it can. > > > By the end of this year I'll get a new desktop box based on Intels new > > > Sandy Bridge-E architecture with plenty of memory. If the colleague who > > > developed the code is willing performing some benchmarks on the same > > > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > > > recent Suse. For FreeBSD I intent also to look for performance with both > > > different schedulers available. > > > > > These observations are not scientific, but I have a CPU from AMD with > 6 cores (AMD Phenom(tm) II X6 1090T Processor). > > My simple test was ``make buildkernel'' while watching the core usage with > gkrellm. > > With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. > I've never seen any value above 97% with gkrellm. > > With SCHED_ULE I never saw all 6 cores loaded this heavily. Usually > 2 or more cores were at or below 90%. Not really that significant, but > still a noticeable difference in apparent scheduling behavior. Whether > the observed difference is due to some change in data from the kernel to > gkrellm is beyond me. > > -- > Gary Jennejohn > _____________________________________________ > > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > -- Gary Jennejohn From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 16:48:59 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D3F4106567A; Mon, 12 Dec 2011 16:48:59 +0000 (UTC) (envelope-from gljennjohn@googlemail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id A03D38FC14; Mon, 12 Dec 2011 16:48:58 +0000 (UTC) Received: by eekc50 with SMTP id c50so3384185eek.13 for ; Mon, 12 Dec 2011 08:48:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=date:from:to:cc:subject:message-id:in-reply-to:references:reply-to :x-mailer:mime-version:content-type:content-transfer-encoding; bh=QFQyElN+lsakH/hXyvqde4RifLlcVgBZTDRMOHlFA6E=; b=L49yFzALu8AFiSjmnq5Zoxakto8S8RZJ5uihrtPrGtPlKoedDQdm0FXoIuWxq91kbJ x1fEYqXOLpCrAM4S8Kku5agZBB0aRgPTOrDIo0G9jd3liNL+dgZMrdNVsf66vaXA6D1P 2/uSipRpqUkwxpwUmlgeRdrYxltzRBNO1Yj+g= Received: by 10.14.4.154 with SMTP id 26mr2818946eej.36.1323708537575; Mon, 12 Dec 2011 08:48:57 -0800 (PST) Received: from ernst.jennejohn.org (p578E15DC.dip.t-dialin.net. [87.142.21.220]) by mx.google.com with ESMTPS id 39sm12128785eei.1.2011.12.12.08.48.55 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 12 Dec 2011 08:48:56 -0800 (PST) Date: Mon, 12 Dec 2011 17:48:54 +0100 From: Gary Jennejohn To: mdf@FreeBSD.org Message-ID: <20111212174854.4f6ea14c@ernst.jennejohn.org> In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <20111212163221.33d0b8a2@ernst.jennejohn.org> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Mon, 12 Dec 2011 23:06:35 +0000 Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Vincent Hoffman Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: gljennjohn@googlemail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 16:48:59 -0000 On Mon, 12 Dec 2011 08:04:37 -0800 mdf@FreeBSD.org wrote: > On Mon, Dec 12, 2011 at 7:32 AM, Gary Jennejohn > wrote: > > On Mon, 12 Dec 2011 15:13:00 +0000 > > Vincent Hoffman wrote: > > > >> > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> On 12/12/2011 13:47, O. Hartmann wrote: > >> > > >> >> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >> >> issue. And yes, there are cases where SCHED_ULE shows much better > >> >> performance then SCHED_4BSD. [...] > >> > > >> > Do we have any proof at hand for such cases where SCHED_ULE performs > >> > much better than SCHED_4BSD? Whenever the subject comes up, it is > >> > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > >> > 2. But in the end I see here contradictionary statements. People > >> > complain about poor performance (especially in scientific environments), > >> > and other give contra not being the case. > >> It all a little old now but some if the stuff in > >> http://people.freebsd.org/~kris/scaling/ > >> covers improvements that were seen. > >> > >> http://jeffr-tech.livejournal.com/5705.html > >> shows a little too, reading though Jeffs blog is worth it as it has some > >> interesting stuff on SHED_ULE. > >> > >> I thought there were some more benchmarks floating round but cant find > >> any with a quick google. > >> > >> > >> Vince > >> > >> > > >> > Within our department, we developed a highly scalable code for planetary > >> > science purposes on imagery. It utilizes present GPUs via OpenCL if > >> > present. Otherwise it grabs as many cores as it can. > >> > By the end of this year I'll get a new desktop box based on Intels new > >> > Sandy Bridge-E architecture with plenty of memory. If the colleague who > >> > developed the code is willing performing some benchmarks on the same > >> > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > >> > recent Suse. For FreeBSD I intent also to look for performance with both > >> > different schedulers available. > >> > > > > > These observations are not scientific, but I have a CPU from AMD with > > 6 cores (AMD Phenom(tm) II X6 1090T Processor). > > > > My simple test was ``make buildkernel'' while watching the core usage with > > gkrellm. > > > > With SCHED_4BSD all 6 cores are loaded to 97% during the build phase. > > I've never seen any value above 97% with gkrellm. > > > > With SCHED_ULE I never saw all 6 cores loaded this heavily.  Usually > > 2 or more cores were at or below 90%.  Not really that significant, but > > still a noticeable difference in apparent scheduling behavior.  Whether > > the observed difference is due to some change in data from the kernel to > > gkrellm is beyond me. > > SCHED_ULE is much sloppier about calculating which thread used a > timeslice -- unless the timeslice went 100% to a thread, the fraction > it used may get attributed elsewhere. So top's reporting of thread > usage is not a useful metric. Total buildworld time is, potentially. > I suspect you're right since the buildworld time, a much better test, was pretty much the same with 4BSD and ULE. -- Gary Jennejohn From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 17:06:07 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0371106564A; Mon, 12 Dec 2011 17:06:07 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id 8DE578FC12; Mon, 12 Dec 2011 17:06:07 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBCH64v5074234; Mon, 12 Dec 2011 09:06:04 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBCH64Fe074233; Mon, 12 Dec 2011 09:06:04 -0800 (PST) (envelope-from sgk) Date: Mon, 12 Dec 2011 09:06:04 -0800 From: Steve Kargl To: Bruce Cran Message-ID: <20111212170604.GA74044@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE6295B.3020308@cran.org.uk> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Mon, 12 Dec 2011 23:06:47 +0000 Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 17:06:07 -0000 On Mon, Dec 12, 2011 at 04:18:35PM +0000, Bruce Cran wrote: > On 12/12/2011 15:51, Steve Kargl wrote: > >This comes up every 9 months or so, and must be approaching FAQ > >status. In a HPC environment, I recommend 4BSD. Depending on the > >workload, ULE can cause a severe increase in turn around time when > >doing already long computations. If you have an MPI application, > >simply launching greater than ncpu+1 jobs can show the problem. PS: > >search the list archives for "kargl and ULE". > > This isn't something that can be fixed by tuning ULE? For example for > desktop applications kern.sched.preempt_thresh should be set to 224 from > its default. I'm wondering if the installer should ask people what the > typical use will be, and tune the scheduler appropriately. > Tuning kern.sched.preempt_thresh did not seem to help for my workload. My code is a classic master-slave OpenMPI application where the master runs on one node and all cpu-bound slaves are sent to a second node. If I send send ncpu+1 jobs to the 2nd node with ncpu's, then ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The last two jobs are assigned to the ncpu'th cpu, and these ping-pong on the this cpu. AFAICT, it is a cpu affinity issue, where ULE is trying to keep each job associated with its initially assigned cpu. While one might suggest that starting ncpu+1 jobs is not prudent, my example is just that. It is an example showing that ULE has performance issues. So, I now can start only ncpu jobs on each node in the cluster and send emails to all other users to not use those node, or use 4BSD and not worry about loading issues. -- Steve From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 18:50:14 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C6341065675; Mon, 12 Dec 2011 18:50:14 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3066C8FC16; Mon, 12 Dec 2011 18:50:14 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id D06F946B3C; Mon, 12 Dec 2011 13:50:13 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 42D2DB961; Mon, 12 Dec 2011 13:50:13 -0500 (EST) From: John Baldwin To: freebsd-current@freebsd.org Date: Mon, 12 Dec 2011 13:50:11 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <4EE1EAFE.3070408@m5p.com> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> In-Reply-To: <20111212170604.GA74044@troutmask.apl.washington.edu> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201112121350.11784.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 12 Dec 2011 13:50:13 -0500 (EST) X-Mailman-Approved-At: Mon, 12 Dec 2011 23:07:02 +0000 Cc: Bruce Cran , "O. Hartmann" , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Steve Kargl Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 18:50:14 -0000 On Monday, December 12, 2011 12:06:04 pm Steve Kargl wrote: > On Mon, Dec 12, 2011 at 04:18:35PM +0000, Bruce Cran wrote: > > On 12/12/2011 15:51, Steve Kargl wrote: > > >This comes up every 9 months or so, and must be approaching FAQ > > >status. In a HPC environment, I recommend 4BSD. Depending on the > > >workload, ULE can cause a severe increase in turn around time when > > >doing already long computations. If you have an MPI application, > > >simply launching greater than ncpu+1 jobs can show the problem. PS: > > >search the list archives for "kargl and ULE". > > > > This isn't something that can be fixed by tuning ULE? For example for > > desktop applications kern.sched.preempt_thresh should be set to 224 from > > its default. I'm wondering if the installer should ask people what the > > typical use will be, and tune the scheduler appropriately. > > > > Tuning kern.sched.preempt_thresh did not seem to help for > my workload. My code is a classic master-slave OpenMPI > application where the master runs on one node and all > cpu-bound slaves are sent to a second node. If I send > send ncpu+1 jobs to the 2nd node with ncpu's, then > ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The > last two jobs are assigned to the ncpu'th cpu, and > these ping-pong on the this cpu. AFAICT, it is a cpu > affinity issue, where ULE is trying to keep each job > associated with its initially assigned cpu. > > While one might suggest that starting ncpu+1 jobs > is not prudent, my example is just that. It is an > example showing that ULE has performance issues. > So, I now can start only ncpu jobs on each node > in the cluster and send emails to all other users > to not use those node, or use 4BSD and not worry > about loading issues. This is a case where 4BSD's naive algorithm will spread out the load more evenly because all the threads are on a single, shared queue and each CPU just grabs the head of the queue when it finishes a timeslice. ULE always assigns threads to a single CPU (even if they aren't pinned to a single CPU using cpuset, etc.) and then tries to balance the load across cores later, but I believe in this case it's rebalancer won't have anything to really do as no matter what it does with the N+1 job it's going to be sharing a CPU with another job. -- John Baldwin From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 19:26:38 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 327C0106566B; Mon, 12 Dec 2011 19:26:38 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id E4EEC8FC17; Mon, 12 Dec 2011 19:26:37 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBCJQbTv087779; Mon, 12 Dec 2011 11:26:37 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBCJQbD3087778; Mon, 12 Dec 2011 11:26:37 -0800 (PST) (envelope-from sgk) Date: Mon, 12 Dec 2011 11:26:37 -0800 From: Steve Kargl To: Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Message-ID: <20111212192637.GA87729@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> <20111212190330.GA69380@sysmon.tcworks.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111212190330.GA69380@sysmon.tcworks.net> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Mon, 12 Dec 2011 23:07:18 +0000 Cc: Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 19:26:38 -0000 On Mon, Dec 12, 2011 at 01:03:30PM -0600, Scott Lambert wrote: > On Mon, Dec 12, 2011 at 09:06:04AM -0800, Steve Kargl wrote: > > Tuning kern.sched.preempt_thresh did not seem to help for > > my workload. My code is a classic master-slave OpenMPI > > application where the master runs on one node and all > > cpu-bound slaves are sent to a second node. If I send > > send ncpu+1 jobs to the 2nd node with ncpu's, then > > ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The > > last two jobs are assigned to the ncpu'th cpu, and > > these ping-pong on the this cpu. AFAICT, it is a cpu > > affinity issue, where ULE is trying to keep each job > > associated with its initially assigned cpu. > > > > While one might suggest that starting ncpu+1 jobs > > is not prudent, my example is just that. It is an > > example showing that ULE has performance issues. > > So, I now can start only ncpu jobs on each node > > in the cluster and send emails to all other users > > to not use those node, or use 4BSD and not worry > > about loading issues. > > Does it meet your expectations if you start (j modulo ncpu) = 0 > jobs on a node? > I've never tried to launch more than ncpu + 1 (or + 2) jobs. I suppose at the time I was investigating the issue, it was determined that 4BSD allowed me to get my work done in a more timely manner. So, I took the path of least resistance. -- Steve From owner-freebsd-performance@FreeBSD.ORG Mon Dec 12 23:50:36 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73169106566B for ; Mon, 12 Dec 2011 23:50:36 +0000 (UTC) (envelope-from dieterbsd@engineer.com) Received: from mailout-us.mail.com (mailout-us.gmx.com [74.208.5.67]) by mx1.freebsd.org (Postfix) with SMTP id 1BBB88FC08 for ; Mon, 12 Dec 2011 23:50:36 +0000 (UTC) Received: (qmail 26511 invoked by uid 0); 12 Dec 2011 23:50:35 -0000 Received: from 67.206.184.126 by rms-us005.v300.gmx.net with HTTP Content-Type: text/plain; charset="utf-8" Date: Mon, 12 Dec 2011 18:50:33 -0500 From: "Dieter BSD" Message-ID: <20111212235034.218250@gmx.com> MIME-Version: 1.0 To: freebsd-performance@freebsd.org,freebsd-fs@freebsd.org X-Authenticated: #74169980 X-Flags: 0001 X-Mailer: GMX.com Web Mailer x-registered: 0 Content-Transfer-Encoding: 8bit X-GMX-UID: HfaUbiU03zOlNR3dAHAhLPF+IGRvbwBx Cc: Subject: Maximum blocksize for FFS? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Dec 2011 23:50:36 -0000 Many recent disks have a 4KiB sector size, so newfs's default 2KiB frag size seems suboptimal for these drives. Newfs's man page states: "The optimal block:fragment ratio is 8:1. Other ratios are possible, but are not recommended, and may produce poor results."  (It is not clear to me what the 8:1 ratio optimizes, or exactly what poor results one should expect with a different ratio?) Thus one would logically think of using 32 KiB blocksize 4KiB fragsize at a minimum with these drives. But, from a discussion in 2009: Bruce Evans wrote: > Any block size above the default (16K) tends to thrash and fragment buffer > cache virtual memory.  This is obviously a good pessimization with lots of > small files, and apparently, as you have found, it is a good pessimization > with a few large files too.  I think severe fragmentation can easily take > several seconds to recover from.  The worst case for causing fragmentaion > is probably a mixture of small and large files. This caused a *severe* performance problem and I was forced to reduce to reduce my blocksize to 16KiB.  :-( For data filesystems with large files (multi GB), there are many advantages to using large blocksizes such as less space wasted on bookkeeping, and faster fsck times. So I'm wondering if this buffer cache virtual memory thrashing/fragmenting problem has been fixed?  Is it safe to use 64KiB/8KiB yet?  IIRC I've read concerns about thrashing/fragmenting due to different filesystems having different sizes, say one filesystem being 16K/2K and another being 64k/8K? Also, has the "bug in vfs_bio.c" been fixed? (64KiB blocksize can hang the system) Any other gottchas? From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 00:04:06 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 629391065676; Tue, 13 Dec 2011 00:04:06 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 150A08FC1C; Tue, 13 Dec 2011 00:04:05 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RaFbr-000792-2w>; Tue, 13 Dec 2011 00:48:39 +0100 Received: from e178008161.adsl.alicedsl.de ([85.178.8.161] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RaFbq-0005wx-U2>; Tue, 13 Dec 2011 00:48:39 +0100 Message-ID: <4EE692D6.5010208@zedat.fu-berlin.de> Date: Tue, 13 Dec 2011 00:48:38 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Steve Kargl References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> In-Reply-To: <20111212170604.GA74044@troutmask.apl.washington.edu> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigBAD169A868B0E5A48D5A4634" X-Originating-IP: 85.178.8.161 Cc: Bruce Cran , "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 00:04:06 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBAD169A868B0E5A48D5A4634 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/12/11 18:06, Steve Kargl wrote: > On Mon, Dec 12, 2011 at 04:18:35PM +0000, Bruce Cran wrote: >> On 12/12/2011 15:51, Steve Kargl wrote: >>> This comes up every 9 months or so, and must be approaching FAQ=20 >>> status. In a HPC environment, I recommend 4BSD. Depending on the=20 >>> workload, ULE can cause a severe increase in turn around time when=20 >>> doing already long computations. If you have an MPI application,=20 >>> simply launching greater than ncpu+1 jobs can show the problem. PS:=20 >>> search the list archives for "kargl and ULE".=20 >> >> This isn't something that can be fixed by tuning ULE? For example for = >> desktop applications kern.sched.preempt_thresh should be set to 224 fr= om=20 >> its default. I'm wondering if the installer should ask people what the= =20 >> typical use will be, and tune the scheduler appropriately. >> Is the tuning of kern.sched.preempt_thresh and a proper method of estimating its correct value for the intended to use workload documented in the manpages, maybe tuning()? I find it hard to crawl a lot of pros and cons of mailing lists for evaluating a correct value of this, seemingly, important tunable. >=20 > Tuning kern.sched.preempt_thresh did not seem to help for > my workload. My code is a classic master-slave OpenMPI > application where the master runs on one node and all > cpu-bound slaves are sent to a second node. If I send > send ncpu+1 jobs to the 2nd node with ncpu's, then=20 > ncpu-1 jobs are assigned to the 1st ncpu-1 cpus. The > last two jobs are assigned to the ncpu'th cpu, and=20 > these ping-pong on the this cpu. AFAICT, it is a cpu > affinity issue, where ULE is trying to keep each job > associated with its initially assigned cpu. >=20 > While one might suggest that starting ncpu+1 jobs > is not prudent, my example is just that. It is an > example showing that ULE has performance issues.=20 > So, I now can start only ncpu jobs on each node > in the cluster and send emails to all other users > to not use those node, or use 4BSD and not worry > about loading issues. >=20 --------------enigBAD169A868B0E5A48D5A4634 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO5pLWAAoJEOgBcD7A/5N86FIIAMlp2MmSfYGAw+Gqn5MuN/s1 VxWt+47R+tii3x2I5rvjigs2+c5BbMhQ5B/+LS1qU8OspeAwWcvqYnXCXwKs7kUo FG+8mmdyVaqt9s1hoh/W4tHgDgL/DCMxwkIfS3yVubjqOltDo7npcre7sMoUaEjL lv0ySiLArwHbnD4mdrC3gJz/fW0enmNOl9wGYWWcUPcDdJ5XdYMSfSGk0W6bpSgA ewDaoPtz1jh/CkLAVH59/cxcHowtsM9YcrdTOPKOIAI9amNChlvtuv8Sv8g2LC9e RhgNHCE6RKVqAIpyIZLTFZ6pUfTtQeI6CtqWHDDAvhYAUEZxZmBDErazPkkirWQ= =prJ+ -----END PGP SIGNATURE----- --------------enigBAD169A868B0E5A48D5A4634-- From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 00:15:17 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D9921065672; Tue, 13 Dec 2011 00:15:17 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id E32588FC16; Tue, 13 Dec 2011 00:15:16 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 64AEAE6209; Tue, 13 Dec 2011 00:15:15 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=gJC95oEqPFyd PxKhOGrYzUyGTDw=; b=lxgw94TUBeTsBgWu2YSPBvEpRbjQcuqTJny+hvnZIiFU tHpeqewHNhRiny1zu6kxuMq9MtveVpYMV4ARIwScFH5LK/wbqs3do4spXmz+aKkU 53to5Y0Rb5TCj5DjChXujp47CaMhTdshe6DkZOlXc6EOfhSTZZzvq5AgNt8lf64= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=Mq2NRg aaRj5CQxemqhNyn9XyA4O4PRCSn1UIY2zSw8JCM6SkjRIxcya2ZbuKOMw6ShJ5Ce WDDQnMskQEeJF3RIQ+eyWgORWCB3/3TW1oi305p5LAIVWxNAVOcnAbBwhaa+6XMP KmyZZwAc3ftWGIVyez2HIur75sxN5zkzmvenI= Received: from [192.168.1.120] (188-220-36-32.zone11.bethere.co.uk [188.220.36.32]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id D898DE6200; Tue, 13 Dec 2011 00:15:14 +0000 (GMT) Message-ID: <4EE69911.9040201@cran.org.uk> Date: Tue, 13 Dec 2011 00:15:13 +0000 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE6295B.3020308@cran.org.uk> <20111212170604.GA74044@troutmask.apl.washington.edu> <4EE692D6.5010208@zedat.fu-berlin.de> In-Reply-To: <4EE692D6.5010208@zedat.fu-berlin.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Steve Kargl Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 00:15:17 -0000 On 12/12/2011 23:48, O. Hartmann wrote: > Is the tuning of kern.sched.preempt_thresh and a proper method of > estimating its correct value for the intended to use workload > documented in the manpages, maybe tuning()? I find it hard to crawl a > lot of pros and cons of mailing lists for evaluating a correct value > of this, seemingly, important tunable. Note that I said "for example" :) I was suggesting that there may be sysctl's that can be tweaked to improve performance. -- Bruce Cran From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 00:29:15 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id B754E1065672; Tue, 13 Dec 2011 00:29:15 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 0F52614FAE3; Tue, 13 Dec 2011 00:29:15 +0000 (UTC) Message-ID: <4EE69C5A.3090005@FreeBSD.org> Date: Mon, 12 Dec 2011 16:29:14 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 13 Dec 2011 01:52:56 +0000 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 00:29:15 -0000 On 12/12/2011 05:47, O. Hartmann wrote: > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? I complained about poor interactive performance of ULE in a desktop environment for years. I had numerous people try to help, including Jeff, with various tunables, dtrace'ing, etc. The cause of the problem was never found. I switched to 4BSD, problem gone. This is on 2 separate systems with core 2 duos. hth, Doug -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 08:40:54 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 60E04106564A; Tue, 13 Dec 2011 08:40:54 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id 0F9278FC08; Tue, 13 Dec 2011 08:40:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=02qllFexF+QvnW1q5ymUZZxjyYi7Er87q5EPTv3rxFA=; b=kRx4fPERvEuj3vMPAbWT5xSfgRyH2ufNdtJmGOjw62bi+Al4rw6lEJxT0y8LkWCq0Rzzk4AH93ZQ2c609v2pQ3bg9iFtYEZ/9k2GKJOhJIkY4iz6XeLHmyFIZ2fuPILkBi2udCx4NZfLQedqEtdLvWNpaDxsm73tgGZCCH0wei4=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1RaNut-000G89-UB ; Tue, 13 Dec 2011 10:40:52 +0200 Date: Tue, 13 Dec 2011 10:40:48 +0200 From: Ivan Klymenko To: Doug Barton Message-ID: <20111213104048.40f3e3de@nonamehost.> In-Reply-To: <4EE69C5A.3090005@FreeBSD.org> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 08:40:54 -0000 > On 12/12/2011 05:47, O. Hartmann wrote: > > Do we have any proof at hand for such cases where SCHED_ULE performs > > much better than SCHED_4BSD? > > I complained about poor interactive performance of ULE in a desktop > environment for years. I had numerous people try to help, including > Jeff, with various tunables, dtrace'ing, etc. The cause of the problem > was never found. > > I switched to 4BSD, problem gone. > > This is on 2 separate systems with core 2 duos. > > > hth, > > Doug > If the algorithm ULE does not contain problems - it means the problem has Core2Duo, or in a piece of code that uses the ULE scheduler. I already wrote in a mailing list that specifically in my case (Core2Duo) partially helps the following patch: --- sched_ule.c.orig 2011-11-24 18:11:48.000000000 +0200 +++ sched_ule.c 2011-12-10 22:47:08.000000000 +0200 @@ -794,7 +794,8 @@ * 1.5 * balance_interval. */ balance_ticks = max(balance_interval / 2, 1); - balance_ticks += random() % balance_interval; +// balance_ticks += random() % balance_interval; + balance_ticks += ((int)random()) % balance_interval; if (smp_started == 0 || rebalance == 0) return; tdq = TDQ_SELF(); @@ -2118,13 +2119,21 @@ struct td_sched *ts; THREAD_LOCK_ASSERT(td, MA_OWNED); + if (td->td_pri_class & PRI_FIFO_BIT) + return; + ts = td->td_sched; + /* + * We used up one time slice. + */ + if (--ts->ts_slice > 0) + return; tdq = TDQ_SELF(); #ifdef SMP /* * We run the long term load balancer infrequently on the first cpu. */ - if (balance_tdq == tdq) { - if (balance_ticks && --balance_ticks == 0) + if (balance_ticks && --balance_ticks == 0) { + if (balance_tdq == tdq) sched_balance(); } #endif @@ -2144,9 +2153,6 @@ if (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) tdq->tdq_ridx = tdq->tdq_idx; } - ts = td->td_sched; - if (td->td_pri_class & PRI_FIFO_BIT) - return; if (PRI_BASE(td->td_pri_class) == PRI_TIMESHARE) { /* * We used a tick; charge it to the thread so @@ -2157,11 +2163,6 @@ sched_priority(td); } /* - * We used up one time slice. - */ - if (--ts->ts_slice > 0) - return; - /* * We're out of time, force a requeue at userret(). */ ts->ts_slice = sched_slice; and refusal to use options FULL_PREEMPTION But no one has unsubscribed to my letter, my patch helps or not in the case of Core2Duo... There is a suspicion that the problems stem from the sections of code associated with the SMP... Maybe I'm in something wrong, but I want to help in solving this problem ... From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 09:53:23 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B8FA1065670 for ; Tue, 13 Dec 2011 09:53:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 1FA458FC16 for ; Tue, 13 Dec 2011 09:53:22 +0000 (UTC) Received: from alf.home (alf.kiev.zoral.com.ua [10.1.1.177]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id pBD9Ewnh076089 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 13 Dec 2011 11:14:58 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: from alf.home (kostik@localhost [127.0.0.1]) by alf.home (8.14.5/8.14.5) with ESMTP id pBD9Evt9038621; Tue, 13 Dec 2011 11:14:57 +0200 (EET) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by alf.home (8.14.5/8.14.5/Submit) id pBD9EvLw038620; Tue, 13 Dec 2011 11:14:57 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: alf.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 13 Dec 2011 11:14:57 +0200 From: Kostik Belousov To: Dieter BSD Message-ID: <20111213091457.GU50300@deviant.kiev.zoral.com.ua> References: <20111212235034.218250@gmx.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MzUVogtTSRhL5bvu" Content-Disposition: inline In-Reply-To: <20111212235034.218250@gmx.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-fs@freebsd.org, freebsd-performance@freebsd.org Subject: Re: Maximum blocksize for FFS? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 09:53:23 -0000 --MzUVogtTSRhL5bvu Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Dec 12, 2011 at 06:50:33PM -0500, Dieter BSD wrote: > Many recent disks have a 4KiB sector size, so newfs's default > 2KiB frag size seems suboptimal for these drives. Newfs's man > page states: "The optimal block:fragment ratio is 8:1. Other > ratios are possible, but are not recommended, and may produce > poor results." =9A(It is not clear to me what the 8:1 ratio optimizes, > or exactly what poor results one should expect with a different ratio?) > Thus one would logically think of using 32 KiB blocksize 4KiB fragsize > at a minimum with these drives. >=20 > But, from a discussion in 2009: >=20 > Bruce Evans wrote: > > Any block size above the default (16K) tends to thrash and fragment buf= fer > > cache virtual memory. =9AThis is obviously a good pessimization with lo= ts of > > small files, and apparently, as you have found, it is a good pessimizat= ion > > with a few large files too. =9AI think severe fragmentation can easily = take > > several seconds to recover from. =9AThe worst case for causing fragment= aion > > is probably a mixture of small and large files. >=20 > This caused a *severe* performance problem and I was forced to reduce to > reduce my blocksize to 16KiB. =9A:-( >=20 > For data filesystems with large files (multi GB), there are many advantag= es > to using large blocksizes such as less space wasted on bookkeeping, > and faster fsck times. >=20 > So I'm wondering if this buffer cache virtual memory thrashing/fragmenting > problem has been fixed? =9AIs it safe to use 64KiB/8KiB yet? =9AIIRC I've > read concerns about thrashing/fragmenting due to different filesystems > having different sizes, say one filesystem being 16K/2K and another being > 64k/8K? >=20 > Also, has the "bug in vfs_bio.c" been fixed? (64KiB blocksize can > hang the system) >=20 > Any other gottchas? I think the default KVA allocated for the buffer cache might be too small for the 64KB blocks. The only known issue that prevented defragmentation from working was fixed in r189878. I did not see further reports of the hangs caused by fragmented buffer KVA. --MzUVogtTSRhL5bvu Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iEYEARECAAYFAk7nF5EACgkQC3+MBN1Mb4gXLQCfc5Avzx04azOdEpi+0xfQ+Ufz EWMAoJhCmRlybh4j1kgdqK88tX+aWEAN =lIRB -----END PGP SIGNATURE----- --MzUVogtTSRhL5bvu-- From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 10:49:41 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 860901065675; Tue, 13 Dec 2011 10:49:41 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 23BAA8FC15; Tue, 13 Dec 2011 10:49:40 +0000 (UTC) Received: by vcbfk1 with SMTP id fk1so7696697vcb.13 for ; Tue, 13 Dec 2011 02:49:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=ZX6KqOeJ20evUcl6Cqd6lqgE4dxILwyIbQ+sbvlpRng=; b=b+if3omSbxdzMRidkCt2TljuSzNNZXoCMYHDOhRUqjArbxZChUgwSBPgUDyAXayK3l vm2mZh0iQVBnmIV7x3GBxWJIcbRmuvH4N/a12Bcy+nSeYAIL2Rf56pEYgNlSX71d7p8E rjngzQjC5WO5c3vdJWVU8Hlj4onsk3Kb7JUOE= MIME-Version: 1.0 Received: by 10.220.230.67 with SMTP id jl3mr1004355vcb.60.1323771768334; Tue, 13 Dec 2011 02:22:48 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.26.50 with HTTP; Tue, 13 Dec 2011 02:22:48 -0800 (PST) In-Reply-To: <20111213090051.GA3339@vniz.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213090051.GA3339@vniz.net> Date: Tue, 13 Dec 2011 02:22:48 -0800 X-Google-Sender-Auth: Z0I2tEWuW7JTpBYJ2eEndqbOV5U Message-ID: From: Adrian Chadd To: Andrey Chernov , Ivan Klymenko , Doug Barton , "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 10:49:41 -0000 On 13 December 2011 01:00, Andrey Chernov wrote: >> If the algorithm ULE does not contain problems - it means the problem >> has Core2Duo, or in a piece of code that uses the ULE scheduler. > > I observe ULE interactivity slowness even on single core machine (Pentium > 4) in very visible places, like 'ps ax' output stucks in the middle by ~1 > second. When I switch back to SHED_4BSD, all slowness is gone. Are you able to provide KTR traces of the scheduler results? Something that can be fed to schedgraph? Adrian From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 07:49:32 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6E3D106564A for ; Tue, 13 Dec 2011 07:49:32 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta05.westchester.pa.mail.comcast.net (qmta05.westchester.pa.mail.comcast.net [76.96.62.48]) by mx1.freebsd.org (Postfix) with ESMTP id 960BF8FC14 for ; Tue, 13 Dec 2011 07:49:32 +0000 (UTC) Received: from omta24.westchester.pa.mail.comcast.net ([76.96.62.76]) by qmta05.westchester.pa.mail.comcast.net with comcast id 8Xb21i0031ei1Bg55XcJ4r; Tue, 13 Dec 2011 07:36:18 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta24.westchester.pa.mail.comcast.net with comcast id 8XcG1i00E1t3BNj3kXcHDk; Tue, 13 Dec 2011 07:36:18 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 6FAD3102C1A; Mon, 12 Dec 2011 23:36:15 -0800 (PST) Date: Mon, 12 Dec 2011 23:36:15 -0800 From: Jeremy Chadwick To: "O. Hartmann" Message-ID: <20111213073615.GA69641@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE6060D.5060201@mail.zedat.fu-berlin.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Tue, 13 Dec 2011 13:12:48 +0000 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 07:49:32 -0000 On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > > Not fully right, boinc defaults to run on idprio 31 so this isn't an > > issue. And yes, there are cases where SCHED_ULE shows much better > > performance then SCHED_4BSD. [...] > > Do we have any proof at hand for such cases where SCHED_ULE performs > much better than SCHED_4BSD? Whenever the subject comes up, it is > mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > 2. But in the end I see here contradictionary statements. People > complain about poor performance (especially in scientific environments), > and other give contra not being the case. > > Within our department, we developed a highly scalable code for planetary > science purposes on imagery. It utilizes present GPUs via OpenCL if > present. Otherwise it grabs as many cores as it can. > By the end of this year I'll get a new desktop box based on Intels new > Sandy Bridge-E architecture with plenty of memory. If the colleague who > developed the code is willing performing some benchmarks on the same > hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > recent Suse. For FreeBSD I intent also to look for performance with both > different schedulers available. This is in no way shape or form the same kind of benchmark as what you're planning to do, but I thought I'd throw it out there for folks to take in as they see fit. I know folks were focused mainly on buildworld. I personally would find it interesting if someone with a higher-end system (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the same test (changing -jX to -j{numofcores} of course). -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | sched_ule =========== - time make -j2 buildworld 1689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io 4565pf+0w - time make -j2 buildkernel 640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w sched_4bsd ============ - time make -j2 buildworld 1662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io 6451pf+0w - time make -j2 buildkernel 638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w software ========== * sched_ule test: FreeBSD 8.2-STABLE, Thu Dec 1 04:37:29 PST 2011 * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST 2011 hardware ========== * Intel Core 2 Duo E8400, 3GHz * Supermicro X7SBA * 8GB ECC RAM (4x2GB), DDR2-800 * Intel 320-series SSD, 80GB: /, swap, /var, /tmp, /usr tuning adjustments / etc. =========================== * Before each scheduler test, system was rebooted to ensure I/O cache and other whatnots were empty * All filesystems stock UFS2 + SU (root is non-SU) * All filesystems had tunefs -t enable applied to them * powerd(8) in use, with two rc.conf variables (per CPU spec): performance_cx_lowest="C2" economy_cx_lowest="C2" * loader.conf kern.maxdsiz="2560M" kern.dfldsiz="2560M" kern.maxssiz="256M" ahci_load="yes" hint.p4tcc.0.disabled="1" hint.acpi_throttle.0.disabled="1" vfs.zfs.arc_max="5120M" * make.conf CPUTYPE?=core2 * src.conf WITHOUT_INET6=true WITHOUT_IPFILTER=true WITHOUT_LIB32=true WITHOUT_KERBEROS=true WITHOUT_PAM_SUPPORT=true WITHOUT_PROFILE=true WITHOUT_SENDMAIL=true * kernel configuration - note: between kernel builds, config was changed to either use SCHED_4BSD or SCHED_ULE respectively. cpu HAMMER ident GENERIC makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_4BSD # Classic BSD scheduler #options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # Debugging options options BREAK_TO_DEBUGGER # Sending a serial BREAK drops to DDB options ALT_BREAK_TO_DEBUGGER # Permit ~ to drop to DDB options KDB # Enable kernel debugger support options KDB_TRACE # Print stack trace automatically on panic options DDB # Support DDB options DDB_NUMSYM # Print numeric value of symbols options GDB # Support remote GDB # CPU frequency control device cpufreq # Bus support. device acpi device pci # Floppy drives device fdc # ATA and ATAPI devices # NOTE: "device ata" is missing because we use the Modular ATA core # to only include the ATA-related drivers we need (e.g. AHCI). device atadisk # ATA disk drives device ataraid # ATA RAID drives device atapicd # ATAPI CDROM drives options ATA_STATIC_ID # Static device numbering # Modular ATA device atacore # Core ATA functionality device ataisa # ISA bus support device atapci # PCI bus support; only generic chipset support device ataahci # AHCI SATA device ataintel # Intel # SCSI peripherals device scbus # SCSI bus (required for SCSI) device da # Direct Access (disks) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) options CAMDEBUG # CAM debugging (camcontrol debug) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc device agp # support several AGP chipsets # Serial (COM) ports device uart # Generic UART driver # PCI Ethernet NICs. device em # Intel PRO/1000 Gigabit Ethernet Family # Wireless NIC cards device wlan # 802.11 support options IEEE80211_DEBUG # enable debug msgs options IEEE80211_AMPDU_AGE # age frames in AMPDU reorder q's device wlan_wep # 802.11 WEP support device wlan_ccmp # 802.11 CCMP support device wlan_tkip # 802.11 TKIP support device wlan_amrr # AMRR transmit rate control algorithm device wlan_acl # MAC Access Control List support # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # USB support device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device usb # USB Bus (required) #device udbp # USB Double Bulk Pipe devices device uhid # "Human Interface Devices" device ukbd # Keyboard device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse # Intel Core/Core2Duo CPU temperature monitoring driver device coretemp # SMBus support, needed for bsdhwmon device smbus device smb device ichsmb # Intel ICH hardware watchdog support device ichwd # pf ALTQ support options ALTQ options ALTQ_CBQ # Class Bases Queueing options ALTQ_RED # Random Early Detection options ALTQ_RIO # RED In/Out options ALTQ_HFSC # Hierarchical Packet Scheduler options ALTQ_CDNR # Traffic conditioner options ALTQ_PRIQ # Priority Queueing options ALTQ_NOPCC # Required for SMP build From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 09:13:26 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C93B1065673; Tue, 13 Dec 2011 09:13:26 +0000 (UTC) (envelope-from ache@vniz.net) Received: from vniz.net (vniz.net [194.87.13.69]) by mx1.freebsd.org (Postfix) with ESMTP id 957758FC14; Tue, 13 Dec 2011 09:13:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by vniz.net (8.14.5/8.14.5) with ESMTP id pBD90rxo003476; Tue, 13 Dec 2011 13:00:53 +0400 (MSK) (envelope-from ache@vniz.net) Received: (from ache@localhost) by localhost (8.14.5/8.14.5/Submit) id pBD90qqf003475; Tue, 13 Dec 2011 13:00:52 +0400 (MSK) (envelope-from ache) Date: Tue, 13 Dec 2011 13:00:51 +0400 From: Andrey Chernov To: Ivan Klymenko Message-ID: <20111213090051.GA3339@vniz.net> Mail-Followup-To: Andrey Chernov , Ivan Klymenko , Doug Barton , "O. Hartmann" , Current FreeBSD , freebsd-stable@FreeBSD.ORG, freebsd-performance@FreeBSD.ORG References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111213104048.40f3e3de@nonamehost.> User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Tue, 13 Dec 2011 13:13:12 +0000 Cc: freebsd-stable@FreeBSD.ORG, "O. Hartmann" , Doug Barton , Current FreeBSD , freebsd-performance@FreeBSD.ORG Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 09:13:26 -0000 On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: > > On 12/12/2011 05:47, O. Hartmann wrote: > > > Do we have any proof at hand for such cases where SCHED_ULE performs > > > much better than SCHED_4BSD? > > > > I complained about poor interactive performance of ULE in a desktop > > environment for years. I had numerous people try to help, including > > Jeff, with various tunables, dtrace'ing, etc. The cause of the problem > > was never found. > > > > I switched to 4BSD, problem gone. > > > > This is on 2 separate systems with core 2 duos. > > > > > > hth, > > > > Doug > > > > If the algorithm ULE does not contain problems - it means the problem > has Core2Duo, or in a piece of code that uses the ULE scheduler. I observe ULE interactivity slowness even on single core machine (Pentium 4) in very visible places, like 'ps ax' output stucks in the middle by ~1 second. When I switch back to SHED_4BSD, all slowness is gone. -- http://ache.vniz.net/ From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 11:13:44 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 505F9106564A; Tue, 13 Dec 2011 11:13:44 +0000 (UTC) (envelope-from ohartman@mail.zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 057EE8FC12; Tue, 13 Dec 2011 11:13:43 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RaQIo-0001nB-Vq>; Tue, 13 Dec 2011 12:13:43 +0100 Received: from telesto.geoinf.fu-berlin.de ([130.133.86.198]) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RaQIo-0006uN-Sb>; Tue, 13 Dec 2011 12:13:42 +0100 Message-ID: <4EE73366.7080304@mail.zedat.fu-berlin.de> Date: Tue, 13 Dec 2011 12:13:42 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Vincent Hoffman References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> In-Reply-To: <4EE619FC.4000601@unsane.co.uk> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig801D5BA68A4D4252A6C8F2B4" X-Originating-IP: 130.133.86.198 X-Mailman-Approved-At: Tue, 13 Dec 2011 13:13:23 +0000 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 11:13:44 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig801D5BA68A4D4252A6C8F2B4 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 12/12/11 16:13, Vincent Hoffman wrote: >=20 > On 12/12/2011 13:47, O. Hartmann wrote: >=20 >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an >>> issue. And yes, there are cases where SCHED_ULE shows much better >>> performance then SCHED_4BSD. [...] >=20 >> Do we have any proof at hand for such cases where SCHED_ULE performs >> much better than SCHED_4BSD? Whenever the subject comes up, it is >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu = > >> 2. But in the end I see here contradictionary statements. People >> complain about poor performance (especially in scientific environments= ), >> and other give contra not being the case. > It all a little old now but some if the stuff in > http://people.freebsd.org/~kris/scaling/ > covers improvements that were seen. >=20 > http://jeffr-tech.livejournal.com/5705.html > shows a little too, reading though Jeffs blog is worth it as it has som= e > interesting stuff on SHED_ULE. >=20 > I thought there were some more benchmarks floating round but cant find > any with a quick google. >=20 >=20 > Vince >=20 >=20 Interesting, there seems to be a much more performant scheduler in 7.0, called SCHED_SMP. I have some faint recalls on that ... where is this beast gone? Oliver --------------enig801D5BA68A4D4252A6C8F2B4 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk7nM2YACgkQU6Ni+wtCKv+SoQD9E1daXYU8i3DtYikG3KoKXf3b J+ujUpCBkPNh4fs1RHUA/RkDAdKThLx4xcV7WgblHwEkkZgyLAaAEbfOz2S/s94I =TMYp -----END PGP SIGNATURE----- --------------enig801D5BA68A4D4252A6C8F2B4-- From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 12:47:10 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 262091065679 for ; Tue, 13 Dec 2011 12:47:10 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.westchester.pa.mail.comcast.net (qmta06.westchester.pa.mail.comcast.net [76.96.62.56]) by mx1.freebsd.org (Postfix) with ESMTP id D6AEE8FC12 for ; Tue, 13 Dec 2011 12:47:09 +0000 (UTC) Received: from omta05.westchester.pa.mail.comcast.net ([76.96.62.43]) by qmta06.westchester.pa.mail.comcast.net with comcast id 8cdv1i0080vyq2s56cnAdZ; Tue, 13 Dec 2011 12:47:10 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta05.westchester.pa.mail.comcast.net with comcast id 8cn81i0191t3BNj3Rcn9GG; Tue, 13 Dec 2011 12:47:10 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 56D23102C19; Tue, 13 Dec 2011 04:47:07 -0800 (PST) Date: Tue, 13 Dec 2011 04:47:07 -0800 From: Jeremy Chadwick To: "O. Hartmann" Message-ID: <20111213124707.GA75440@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE619FC.4000601@unsane.co.uk> <4EE73366.7080304@mail.zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE73366.7080304@mail.zedat.fu-berlin.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Tue, 13 Dec 2011 13:13:46 +0000 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org, Vincent Hoffman Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 12:47:10 -0000 On Tue, Dec 13, 2011 at 12:13:42PM +0100, O. Hartmann wrote: > On 12/12/11 16:13, Vincent Hoffman wrote: > > > > On 12/12/2011 13:47, O. Hartmann wrote: > > > >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >>> issue. And yes, there are cases where SCHED_ULE shows much better > >>> performance then SCHED_4BSD. [...] > > > >> Do we have any proof at hand for such cases where SCHED_ULE performs > >> much better than SCHED_4BSD? Whenever the subject comes up, it is > >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > >> 2. But in the end I see here contradictionary statements. People > >> complain about poor performance (especially in scientific environments), > >> and other give contra not being the case. > > It all a little old now but some if the stuff in > > http://people.freebsd.org/~kris/scaling/ > > covers improvements that were seen. > > > > http://jeffr-tech.livejournal.com/5705.html > > shows a little too, reading though Jeffs blog is worth it as it has some > > interesting stuff on SHED_ULE. > > > > I thought there were some more benchmarks floating round but cant find > > any with a quick google. > > > > > > Vince > > > > > > Interesting, there seems to be a much more performant scheduler in 7.0, > called SCHED_SMP. I have some faint recalls on that ... where is this > beast gone? Boy I sure hope I remember this right. I strongly urge others to correct me where I'm wrong; thanks in advance! The classic scheduler, SCHED_4BSD, was implemented back before there was oxygen. sched_4bsd(4) mentions this. No need to discuss it. Jeff Robertson began working on the "first-generation ULE scheduler" during the days of FreeBSD 5.x (I believe 5.1), and a paper on it was presented at USENIX circa 2003: http://www.usenix.org/event/bsdcon03/tech/full_papers/roberson/roberson.pdf Over the following years, Jeff (and others I assume -- maybe folks like George Neville-Neil and/or Kirk McKusick?) adjusted and tinkered with some of the semantics and models/methods. If I remember right, some of these quirks/fixes were committed. All of this was happening under the scheduler that was then called SCHED_ULE, but it was "ULE 1.0" for lack of better terminology. This scheduler did not perform well, if I remember right, and Jeff was quite honest about that. From this point forward, Jeff began idealising and working on a scheduler which he called SCHED_SMP -- think of it as "ULE 2.0", again, for lack of better terminology. It was different than the existing SCHED_ULE scheduler, hence a different name. Jeff blogged about this in early 2007, using exactly that term ("ULE 2.0"): http://jeffr-tech.livejournal.com/3729.html In mid-2007, prior to FreeBSD 7.0-RELEASE, Jeff announced that effectively he wanted to make SCHED_ULE do what SCHED_SMP did, and provided a patch to SCHED_ULE to accomplish just that: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2007-07/msg00755.html Full thread is here (beware -- many replies): http://unix.derkeiler.com/Mailing-Lists/FreeBSD/current/2007-07/threads.html#00755 The patch mentioned above was merged into HEAD on 2007/07/19. http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sched_ule.c#rev1.202 So in effect, as of 2007/07/19, SCHED_ULE became SCHED_SMP. FreeBSD 7.0-RELEASE was released on 2008/02/27, and the above commit/changes were available at that time as well (meaning: RELENG_7 and RELENG_7_0 at that moment in time should have included the patch from the above paragraph). The document released by Kris Kenneway hinted at those changes and performance improvements: http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf Keep in mind, however, that at that time kernel configuration files (GENERIC, etc.) still defaulted to SCHED_4BSD. The default scheduler in kernel config files (GENERIC, etc.) for i386 and amd64 (not sure about others) was changed in 2007/10/19: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/i386/conf/GENERIC#rev1.475 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/amd64/conf/GENERIC#rev1.485 This was done *prior* to FreeBSD 7.1-RELEASE. So, it first became available as the default scheduler "for the masses" when 7.1-RELEASE came out on 2009/01/05. "All of the answers", in a roundabout and non-user-friendly way, are available by examining the commit history for src/sys/kern/sched_ule.c. It's hard to follow especially given that you have to consider all the releases/branchpoints that took place over time, but: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/kern/sched_ule.c Are we having fun yet? :-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB | From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 13:23:53 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BFC161065673; Tue, 13 Dec 2011 13:23:53 +0000 (UTC) (envelope-from ohartman@mail.zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 725158FC1F; Tue, 13 Dec 2011 13:23:53 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RaSKm-0006OK-Av>; Tue, 13 Dec 2011 14:23:52 +0100 Received: from telesto.geoinf.fu-berlin.de ([130.133.86.198]) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RaSKm-0007tJ-7i>; Tue, 13 Dec 2011 14:23:52 +0100 Message-ID: <4EE751E2.60204@mail.zedat.fu-berlin.de> Date: Tue, 13 Dec 2011 14:23:46 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Steve Kargl References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> In-Reply-To: <20111212155159.GB73597@troutmask.apl.washington.edu> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig1B23B528A575FB784F3E4BE3" X-Originating-IP: 130.133.86.198 X-Mailman-Approved-At: Tue, 13 Dec 2011 13:42:16 +0000 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 13:23:54 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig1B23B528A575FB784F3E4BE3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/12/11 16:51, Steve Kargl wrote: > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: >> >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an >>> issue. And yes, there are cases where SCHED_ULE shows much better >>> performance then SCHED_4BSD. [...] >> >> Do we have any proof at hand for such cases where SCHED_ULE performs >> much better than SCHED_4BSD? Whenever the subject comes up, it is >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu = > >> 2. But in the end I see here contradictionary statements. People >> complain about poor performance (especially in scientific environments= ), >> and other give contra not being the case. >> >> Within our department, we developed a highly scalable code for planeta= ry >> science purposes on imagery. It utilizes present GPUs via OpenCL if >> present. Otherwise it grabs as many cores as it can. >> By the end of this year I'll get a new desktop box based on Intels new= >> Sandy Bridge-E architecture with plenty of memory. If the colleague wh= o >> developed the code is willing performing some benchmarks on the same >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most >> recent Suse. For FreeBSD I intent also to look for performance with bo= th >> different schedulers available. >> >=20 > This comes up every 9 months or so, and must be approaching > FAQ status. >=20 > In a HPC environment, I recommend 4BSD. Depending on > the workload, ULE can cause a severe increase in turn > around time when doing already long computations. If > you have an MPI application, simply launching greater > than ncpu+1 jobs can show the problem. Well, those recommendations should based on "WHY". As the mostly negative experiences with SCHED_ULE in highly computative workloads get allways contradicted by "...but there are workloads that show the opposite ..." this should be shown by more recent benchmarks and explanations than legacy benchmarks from years ago. And, indeed, I highly would recommend having a FAQ or a short note in "tuning" or the handbook in which it is mentioned to use SCHED_4BSD in HPC environments and SCHED_ULE for other workloads (which has to be more specific). It is not an easy task setting up a certain kind of OS for a specific purpose and tuning by crawling the mailing lists. Some notes and hints in the documentation is always a valuable hint and highly appreciated by folks not deep into development. And by the way, I have the deep impression that most of these discussions about the poor performance of SCHED_ULE tend to always end up in a covering up that flaw and the conclusive waste of development. But this is only my personal impression. --------------enig1B23B528A575FB784F3E4BE3 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iF4EAREIAAYFAk7nUegACgkQU6Ni+wtCKv9RzAD9GhKwCryupmWNA64Wp5N7rMzk +WcXFTQXR19FqiE3OkMA/3PRpLaK7SJcAh3PiyujWMGBt1smTuui80KUWTatd5+5 =SLRk -----END PGP SIGNATURE----- --------------enig1B23B528A575FB784F3E4BE3-- From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 15:54:57 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B0C9106564A; Tue, 13 Dec 2011 15:54:57 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id BD1178FC12; Tue, 13 Dec 2011 15:54:56 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBDFsuaL093141; Tue, 13 Dec 2011 07:54:56 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBDFsuJ2093140; Tue, 13 Dec 2011 07:54:56 -0800 (PST) (envelope-from sgk) Date: Tue, 13 Dec 2011 07:54:56 -0800 From: Steve Kargl To: "O. Hartmann" Message-ID: <20111213155456.GA93017@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE751E2.60204@mail.zedat.fu-berlin.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE751E2.60204@mail.zedat.fu-berlin.de> User-Agent: Mutt/1.4.2.3i X-Mailman-Approved-At: Tue, 13 Dec 2011 19:12:35 +0000 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 15:54:57 -0000 On Tue, Dec 13, 2011 at 02:23:46PM +0100, O. Hartmann wrote: > On 12/12/11 16:51, Steve Kargl wrote: > > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > >> > >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an > >>> issue. And yes, there are cases where SCHED_ULE shows much better > >>> performance then SCHED_4BSD. [...] > >> > >> Do we have any proof at hand for such cases where SCHED_ULE performs > >> much better than SCHED_4BSD? Whenever the subject comes up, it is > >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > >> 2. But in the end I see here contradictionary statements. People > >> complain about poor performance (especially in scientific environments), > >> and other give contra not being the case. > >> > >> Within our department, we developed a highly scalable code for planetary > >> science purposes on imagery. It utilizes present GPUs via OpenCL if > >> present. Otherwise it grabs as many cores as it can. > >> By the end of this year I'll get a new desktop box based on Intels new > >> Sandy Bridge-E architecture with plenty of memory. If the colleague who > >> developed the code is willing performing some benchmarks on the same > >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > >> recent Suse. For FreeBSD I intent also to look for performance with both > >> different schedulers available. > >> > > > > This comes up every 9 months or so, and must be approaching > > FAQ status. > > > > In a HPC environment, I recommend 4BSD. Depending on > > the workload, ULE can cause a severe increase in turn > > around time when doing already long computations. If > > you have an MPI application, simply launching greater > > than ncpu+1 jobs can show the problem. > > Well, those recommendations should based on "WHY". As the mostly > negative experiences with SCHED_ULE in highly computative workloads get > allways contradicted by "...but there are workloads that show the > opposite ..." this should be shown by more recent benchmarks and > explanations than legacy benchmarks from years ago. > I have given the WHY in previous discussions of ULE, based on what you call legacy benchmarks. I have not seen any commit to sched_ule.c that would lead me to believe that the performance issues with ULE and cpu-bound numerical codes have been addressed. Repeating the benchmark would be a waste of time. -- Steve From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 21:25:31 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F38F106564A; Tue, 13 Dec 2011 21:25:31 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 2E5C58FC08; Tue, 13 Dec 2011 21:25:31 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBDLPTtq038329; Tue, 13 Dec 2011 16:25:29 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EE7C2BE.9020509@sentex.net> Date: Tue, 13 Dec 2011 16:25:18 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE751E2.60204@mail.zedat.fu-berlin.de> <20111213155456.GA93017@troutmask.apl.washington.edu> In-Reply-To: <20111213155456.GA93017@troutmask.apl.washington.edu> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Current FreeBSD , freebsd-stable@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 21:25:31 -0000 On 12/13/2011 10:54 AM, Steve Kargl wrote: > > I have given the WHY in previous discussions of ULE, based > on what you call legacy benchmarks. I have not seen any > commit to sched_ule.c that would lead me to believe that > the performance issues with ULE and cpu-bound numerical > codes have been addressed. Repeating the benchmark would > be a waste of time. Trying a simple pbzip2 on a large file, the results are pretty consistent through iterations. pbzip2 with 4BSD is barely faster on a file thats 322MB in size. after a reboot, I did a strings bigfile > /dev/null then ran pbzip2 -v xaa -c > /dev/null 7 times If I do a burnP6 in the background, they perform about the same. (from sysutils/cpuburn) eg pbzip2 -v xaa -c > /dev/null Parallel BZIP2 v1.1.6 - by: Jeff Gilchrist [http://compression.ca] [Oct. 30, 2011] (uses libbzip2 by Julian Seward) Major contributions: Yavor Nikolov # CPUs: 4 BWT Block Size: 900 KB File Block Size: 900 KB Maximum Memory: 100 MB ------------------------------------------- File #: 1 of 1 Input Name: xaa Output Name: Input Size: 352404831 bytes Compressing data... Output Size: 50630745 bytes ------------------------------------------- Wall Clock: 18.139342 seconds ULE 18.113204 18.116896 18.123400 18.105894 18.163332 18.139342 18.082888 ULE with burnP6 23.076085 22.003666 21.162987 21.682445 21.935568 23.595781 21.601277 4BSD 17.983395 17.986218 18.009254 18.004312 18.001494 17.997032 4BSD with burnP6 22.215508 21.886459 21.595179 21.361830 21.325351 21.244793 # ministat uleP6 bsdP6 x uleP6 + bsdP6 +------------------------------------------------------------------------------------------------------------------------------------------+ |x + + + + x + x x + x x| | |____|______________MA____________________|M_____________A__________________________________________________| | +------------------------------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 6 21.162987 23.595781 22.003666 22.242755 0.91175566 + 6 21.244793 22.215508 21.595179 21.604853 0.3792413 No difference proven at 95.0% confidence x ule + bsd +------------------------------------------------------------------------------------------------------------------------------------------+ |+ + + + + + x x x x x x x| | |______A___M___| |________________M__A__________________| | +------------------------------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 7 18.082888 18.163332 18.116896 18.120708 0.025468695 + 6 17.983395 18.009254 18.001494 17.996951 0.010248473 Difference at 95.0% confidence -0.123757 +/- 0.024538 -0.68296% +/- 0.135414% (Student's t, pooled s = 0.0200388) hardware is X3450 with 8G of memory. RELENG8 ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 23:39:14 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09B051065670; Tue, 13 Dec 2011 23:39:14 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id 8A7C18FC08; Tue, 13 Dec 2011 23:39:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=5F70nZqRcYgm7bijrHgHyKB7UoBju6OC5/bO8dW/88o=; b=UCCsjgOajR4VEgZZUmFT8w1hirtKquOO5Imc+Y+MxOaCwFvWJHpSRzawb4RG8UtOwO39O8ln9yMHmfUQFZfk4sHuyDdTgiD+e5YJGQ9o6LPDgvP/+XA1/4abE7Fxon5FSIdselgvFv3ZKh+9DCi7ybFm7aRNXgx+H0jDZkIHJv0=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1RabwD-0003M3-1I ; Wed, 14 Dec 2011 01:39:09 +0200 Date: Wed, 14 Dec 2011 01:39:06 +0200 From: Ivan Klymenko To: Jilles Tjoelker Message-ID: <20111214013906.068f69df@nonamehost.> In-Reply-To: <20111213230441.GB42285@stack.nl> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Doug Barton , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Current FreeBSD Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 23:39:14 -0000 =D0=92 Wed, 14 Dec 2011 00:04:42 +0100 Jilles Tjoelker =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: > > If the algorithm ULE does not contain problems - it means the > > problem has Core2Duo, or in a piece of code that uses the ULE > > scheduler. I already wrote in a mailing list that specifically in > > my case (Core2Duo) partially helps the following patch: > > --- sched_ule.c.orig 2011-11-24 18:11:48.000000000 +0200 > > +++ sched_ule.c 2011-12-10 22:47:08.000000000 +0200 > > @@ -794,7 +794,8 @@ > > * 1.5 * balance_interval. > > */ > > balance_ticks =3D max(balance_interval / 2, 1); > > - balance_ticks +=3D random() % balance_interval; > > +// balance_ticks +=3D random() % balance_interval; > > + balance_ticks +=3D ((int)random()) % balance_interval; > > if (smp_started =3D=3D 0 || rebalance =3D=3D 0) > > return; > > tdq =3D TDQ_SELF(); >=20 > This avoids a 64-bit division on 64-bit platforms but seems to have no > effect otherwise. Because this function is not called very often, the > change seems unlikely to help. Yes, this section does not apply to this problem :) Just I posted the latest patch which i using now... >=20 > > @@ -2118,13 +2119,21 @@ > > struct td_sched *ts; > > =20 > > THREAD_LOCK_ASSERT(td, MA_OWNED); > > + if (td->td_pri_class & PRI_FIFO_BIT) > > + return; > > + ts =3D td->td_sched; > > + /* > > + * We used up one time slice. > > + */ > > + if (--ts->ts_slice > 0) > > + return; >=20 > This skips most of the periodic functionality (long term load > balancer, saving switch count (?), insert index (?), interactivity > score update for long running thread) if the thread is not going to > be rescheduled right now. >=20 > It looks wrong but it is a data point if it helps your workload. Yes, I did it for as long as possible to delay the execution of the code in= section: ... #ifdef SMP /* * We run the long term load balancer infrequently on the first cpu. */ if (balance_tdq =3D=3D tdq) { if (balance_ticks && --balance_ticks =3D=3D 0) sched_balance(); } #endif ... >=20 > > tdq =3D TDQ_SELF(); > > #ifdef SMP > > /* > > * We run the long term load balancer infrequently on the > > first cpu. */ > > - if (balance_tdq =3D=3D tdq) { > > - if (balance_ticks && --balance_ticks =3D=3D 0) > > + if (balance_ticks && --balance_ticks =3D=3D 0) { > > + if (balance_tdq =3D=3D tdq) > > sched_balance(); > > } > > #endif >=20 > The main effect of this appears to be to disable the long term load > balancer completely after some time. At some point, a CPU other than > the first CPU (which uses balance_tdq) will set balance_ticks =3D 0, and > sched_balance() will never be called again. >=20 That is, for the same reason as above in the text... > It also introduces a hypothetical race condition because the access to > balance_ticks is no longer restricted to one CPU under a spinlock. >=20 > If the long term load balancer may be causing trouble, try setting > kern.sched.balance_interval to a higher value with unpatched code. I checked it in the first place - but it did not help fix the situation... The impression of malfunction rebalancing... It seems that the thread is passed on to the same core that is loaded and s= o... Perhaps this is a consequence of an incorrect definition of the topology CP= U? >=20 > > @@ -2144,9 +2153,6 @@ > > if > > (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) > > tdq->tdq_ridx =3D tdq->tdq_idx; } > > - ts =3D td->td_sched; > > - if (td->td_pri_class & PRI_FIFO_BIT) > > - return; > > if (PRI_BASE(td->td_pri_class) =3D=3D PRI_TIMESHARE) { > > /* > > * We used a tick; charge it to the thread so > > @@ -2157,11 +2163,6 @@ > > sched_priority(td); > > } > > /* > > - * We used up one time slice. > > - */ > > - if (--ts->ts_slice > 0) > > - return; > > - /* > > * We're out of time, force a requeue at userret(). > > */ > > ts->ts_slice =3D sched_slice; >=20 > > and refusal to use options FULL_PREEMPTION > > But no one has unsubscribed to my letter, my patch helps or not in > > the case of Core2Duo... > > There is a suspicion that the problems stem from the sections of > > code associated with the SMP... > > Maybe I'm in something wrong, but I want to help in solving this > > problem ... >=20 From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 23:42:15 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F12F81065703; Tue, 13 Dec 2011 23:42:14 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm1.ukr.net (fsm1.ukr.net [195.214.192.120]) by mx1.freebsd.org (Postfix) with ESMTP id A00CD8FC1A; Tue, 13 Dec 2011 23:42:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=sw3zgxPAR3QAfGKCe6TuQNxbPW4sLLWcsOBc2vusc8o=; b=M+tGRDGYgsoljjuix/VjsnDBpi6i74Dlzd8dfI6e/B9ADT1z/g07FroZ3i3PrwHMZSKrk5ZMvuPz+hGrXrKXdv4F48onYzXCsj6A6osK4DwY+C1igtpK4mUPbE5VlNiD4pun5cGDHng615clnM6+XbGUMwaZdZXCz/o0vpbLGzo=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm1.ukr.net with esmtpsa ID 1RabzA-000Ofi-Px ; Wed, 14 Dec 2011 01:42:12 +0200 Date: Wed, 14 Dec 2011 01:42:11 +0200 From: Ivan Klymenko To: Marcus Reid Message-ID: <20111214014211.3e108b53@nonamehost.> In-Reply-To: <20111213230215.GA83159@blazingdot.com> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213230215.GA83159@blazingdot.com> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Doug Barton , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Current FreeBSD Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 23:42:15 -0000 =D0=92 Tue, 13 Dec 2011 23:02:15 +0000 Marcus Reid =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On Mon, Dec 12, 2011 at 04:29:14PM -0800, Doug Barton wrote: > > On 12/12/2011 05:47, O. Hartmann wrote: > > > Do we have any proof at hand for such cases where SCHED_ULE > > > performs much better than SCHED_4BSD? > >=20 > > I complained about poor interactive performance of ULE in a desktop > > environment for years. I had numerous people try to help, including > > Jeff, with various tunables, dtrace'ing, etc. The cause of the > > problem was never found. >=20 > The issues that I've seen with ULE on the desktop seem to be caused > by X taking up a steady amount of CPU, and being demoted from being an > "interactive" process. X then becomes the bottleneck for other > processes that would otherwise be "interactive". Try 'renice -20 > ' and see if that makes your problems go away. Why, then X is not a bottleneck when using 4BSD? > Marcus From owner-freebsd-performance@FreeBSD.ORG Wed Dec 14 00:36:32 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17C73106564A; Wed, 14 Dec 2011 00:36:32 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id 9EFAE8FC1A; Wed, 14 Dec 2011 00:36:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=18YzJbAWI+aRJeAk77qSqDa/FDJYKyD0tpp/v1biG78=; b=hU8yO3T7w8C35eMlOLcyxigofcXvWzFwT0yKaGwGu0+CIHOUQ65gVLbMS9S0uYZHPuuCF0dr1ReSYBXAbNGWg4ek9lS9Uf1kCB87IsQAYiE4xYutmpjDX20QLU1txXgmYO/xq6gVxXyGrg+LbeHfSbx/TXN1rF8RP0YafM1EnTg=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1Racpi-000EAS-H6 ; Wed, 14 Dec 2011 02:36:30 +0200 Date: Wed, 14 Dec 2011 02:36:29 +0200 From: Ivan Klymenko To: mdf@FreeBSD.org Message-ID: <20111214023629.3ae8c928@nonamehost.> In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Doug Barton , freebsd-stable@freebsd.org, Tjoelker , "O. Hartmann" , Current FreeBSD , Jilles, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 00:36:32 -0000 =D0=92 Tue, 13 Dec 2011 16:01:56 -0800 mdf@FreeBSD.org =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On Tue, Dec 13, 2011 at 3:39 PM, Ivan Klymenko wrote: > > =D0=92 Wed, 14 Dec 2011 00:04:42 +0100 > > Jilles Tjoelker =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > > >> On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: > >> > If the algorithm ULE does not contain problems - it means the > >> > problem has Core2Duo, or in a piece of code that uses the ULE > >> > scheduler. I already wrote in a mailing list that specifically in > >> > my case (Core2Duo) partially helps the following patch: > >> > --- sched_ule.c.orig =C2=A0 =C2=A0 =C2=A0 =C2=A02011-11-24 18:11:48.= 000000000 +0200 > >> > +++ sched_ule.c =C2=A0 =C2=A0 2011-12-10 22:47:08.000000000 +0200 > >> > @@ -794,7 +794,8 @@ > >> > =C2=A0 =C2=A0 =C2=A0* 1.5 * balance_interval. > >> > =C2=A0 =C2=A0 =C2=A0*/ > >> > =C2=A0 =C2=A0 balance_ticks =3D max(balance_interval / 2, 1); > >> > - =C2=A0 balance_ticks +=3D random() % balance_interval; > >> > +// balance_ticks +=3D random() % balance_interval; > >> > + =C2=A0 balance_ticks +=3D ((int)random()) % balance_interval; > >> > =C2=A0 =C2=A0 if (smp_started =3D=3D 0 || rebalance =3D=3D 0) > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > =C2=A0 =C2=A0 tdq =3D TDQ_SELF(); > >> > >> This avoids a 64-bit division on 64-bit platforms but seems to > >> have no effect otherwise. Because this function is not called very > >> often, the change seems unlikely to help. > > > > Yes, this section does not apply to this problem :) > > Just I posted the latest patch which i using now... > > > >> > >> > @@ -2118,13 +2119,21 @@ > >> > =C2=A0 =C2=A0 struct td_sched *ts; > >> > > >> > =C2=A0 =C2=A0 THREAD_LOCK_ASSERT(td, MA_OWNED); > >> > + =C2=A0 if (td->td_pri_class & PRI_FIFO_BIT) > >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > + =C2=A0 ts =3D td->td_sched; > >> > + =C2=A0 /* > >> > + =C2=A0 =C2=A0* We used up one time slice. > >> > + =C2=A0 =C2=A0*/ > >> > + =C2=A0 if (--ts->ts_slice > 0) > >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > >> This skips most of the periodic functionality (long term load > >> balancer, saving switch count (?), insert index (?), interactivity > >> score update for long running thread) if the thread is not going to > >> be rescheduled right now. > >> > >> It looks wrong but it is a data point if it helps your workload. > > > > Yes, I did it for as long as possible to delay the execution of the > > code in section: ... > > #ifdef SMP > > =C2=A0 =C2=A0 =C2=A0 =C2=A0/* > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 * We run the long term load balancer infreq= uently on the > > first cpu. */ > > =C2=A0 =C2=A0 =C2=A0 =C2=A0if (balance_tdq =3D=3D tdq) { > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (balance_tick= s && --balance_ticks =3D=3D 0) > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0sched_balance(); > > =C2=A0 =C2=A0 =C2=A0 =C2=A0} > > #endif > > ... > > > >> > >> > =C2=A0 =C2=A0 tdq =3D TDQ_SELF(); > >> > =C2=A0#ifdef SMP > >> > =C2=A0 =C2=A0 /* > >> > =C2=A0 =C2=A0 =C2=A0* We run the long term load balancer infrequentl= y on the > >> > first cpu. */ > >> > - =C2=A0 if (balance_tdq =3D=3D tdq) { > >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (balance_ticks && --balance_= ticks =3D=3D 0) > >> > + =C2=A0 if (balance_ticks && --balance_ticks =3D=3D 0) { > >> > + =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (balance_tdq =3D=3D tdq) > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 sched_balance(); > >> > =C2=A0 =C2=A0 } > >> > =C2=A0#endif > >> > >> The main effect of this appears to be to disable the long term load > >> balancer completely after some time. At some point, a CPU other > >> than the first CPU (which uses balance_tdq) will set balance_ticks > >> =3D 0, and sched_balance() will never be called again. > >> > > > > That is, for the same reason as above in the text... > > > >> It also introduces a hypothetical race condition because the > >> access to balance_ticks is no longer restricted to one CPU under a > >> spinlock. > >> > >> If the long term load balancer may be causing trouble, try setting > >> kern.sched.balance_interval to a higher value with unpatched code. > > > > I checked it in the first place - but it did not help fix the > > situation... > > > > The impression of malfunction rebalancing... > > It seems that the thread is passed on to the same core that is > > loaded and so... Perhaps this is a consequence of an incorrect > > definition of the topology CPU? > > > >> > >> > @@ -2144,9 +2153,6 @@ > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if > >> > (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) > >> > tdq->tdq_ridx =3D tdq->tdq_idx; } > >> > - =C2=A0 ts =3D td->td_sched; > >> > - =C2=A0 if (td->td_pri_class & PRI_FIFO_BIT) > >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > =C2=A0 =C2=A0 if (PRI_BASE(td->td_pri_class) =3D=3D PRI_TIMESHARE) { > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0* We used a tick; ch= arge it to the thread so > >> > @@ -2157,11 +2163,6 @@ > >> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sched_priority(td); > >> > =C2=A0 =C2=A0 } > >> > =C2=A0 =C2=A0 /* > >> > - =C2=A0 =C2=A0* We used up one time slice. > >> > - =C2=A0 =C2=A0*/ > >> > - =C2=A0 if (--ts->ts_slice > 0) > >> > - =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return; > >> > - =C2=A0 /* > >> > =C2=A0 =C2=A0 =C2=A0* We're out of time, force a requeue at userret(= ). > >> > =C2=A0 =C2=A0 =C2=A0*/ > >> > =C2=A0 =C2=A0 ts->ts_slice =3D sched_slice; > >> > >> > and refusal to use options FULL_PREEMPTION > >> > But no one has unsubscribed to my letter, my patch helps or not > >> > in the case of Core2Duo... > >> > There is a suspicion that the problems stem from the sections of > >> > code associated with the SMP... > >> > Maybe I'm in something wrong, but I want to help in solving this > >> > problem ... >=20 >=20 > Has anyone experiencing problems tried to set sysctl > kern.sched.steal_thresh=3D1 ? >=20 In my case, the variable kern.sched.steal_thresh and so has the value 1. > I don't remember what our specific problem at $WORK was, perhaps it > was just interrupt threads not getting serviced fast enough, but we've > hard-coded this to 1 and removed the code that sets it in > sched_initticks(). The same effect should be had by setting the > sysctl after a box is up. >=20 > Thanks, > matthew From owner-freebsd-performance@FreeBSD.ORG Wed Dec 14 03:25:18 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 373EB106566B; Wed, 14 Dec 2011 03:25:18 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx06.syd.optusnet.com.au (fallbackmx06.syd.optusnet.com.au [211.29.132.8]) by mx1.freebsd.org (Postfix) with ESMTP id A20CD8FC08; Wed, 14 Dec 2011 03:25:16 +0000 (UTC) Received: from mail15.syd.optusnet.com.au (mail15.syd.optusnet.com.au [211.29.132.196]) by fallbackmx06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id pBE1PJRp009709; Wed, 14 Dec 2011 12:25:19 +1100 Received: from c211-28-227-231.carlnfd1.nsw.optusnet.com.au (c211-28-227-231.carlnfd1.nsw.optusnet.com.au [211.28.227.231]) by mail15.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id pBE1PEwA000923 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 14 Dec 2011 12:25:14 +1100 Date: Wed, 14 Dec 2011 12:25:14 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Ivan Klymenko In-Reply-To: <20111214013906.068f69df@nonamehost.> Message-ID: <20111214115809.T1563@besplex.bde.org> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <20111214013906.068f69df@nonamehost.> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-1963381402-1323825914=:1563" Cc: Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 03:25:18 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-1963381402-1323825914=:1563 Content-Type: TEXT/PLAIN; charset=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Wed, 14 Dec 2011, Ivan Klymenko wrote: > =D0=92 Wed, 14 Dec 2011 00:04:42 +0100 > Jilles Tjoelker =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > >> On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: >>> If the algorithm ULE does not contain problems - it means the >>> problem has Core2Duo, or in a piece of code that uses the ULE >>> scheduler. I already wrote in a mailing list that specifically in >>> my case (Core2Duo) partially helps the following patch: >>> --- sched_ule.c.orig=092011-11-24 18:11:48.000000000 +0200 >>> +++ sched_ule.c=092011-12-10 22:47:08.000000000 +0200 >>> ... >>> @@ -2118,13 +2119,21 @@ >>> =09struct td_sched *ts; >>> >>> =09THREAD_LOCK_ASSERT(td, MA_OWNED); >>> +=09if (td->td_pri_class & PRI_FIFO_BIT) >>> +=09=09return; >>> +=09ts =3D td->td_sched; >>> +=09/* >>> +=09 * We used up one time slice. >>> +=09 */ >>> +=09if (--ts->ts_slice > 0) >>> +=09=09return; >> >> This skips most of the periodic functionality (long term load >> balancer, saving switch count (?), insert index (?), interactivity >> score update for long running thread) if the thread is not going to >> be rescheduled right now. >> >> It looks wrong but it is a data point if it helps your workload. > > Yes, I did it for as long as possible to delay the execution of the code = in section: I don't understand what you are doing here, but recently noticed that the timeslicing in SCHED_4BSD is completely broken. This bug may be a feature. SCHED_4BSD doesn't have its own timeslice counter like ts_slice above. It uses `switchticks' instead. But switchticks hasn't been usable for this purpose since long before SCHED_4BSD started using it for this purpose. switchticks is reset on every context switch, so it is useless for almost all purposes -- any interrupt activity on a non-fast interrupt clobbers it. Removing the check of ts_slice in the above and always returning might give a similar bug to the SCHED_4BSD one. I noticed this while looking for bugs in realtime scheduling. In the above, returning early for PRI_FIFO_BIT also skips most of the periodic functionality. In SCHED_4BSD, returning early is the usual case, so the PRI_FIFO_BIT might as well not be checked, and it is the unusual fifo scheduling case (which is supposed to only apply to realtime priority threads) which has a chance of working as intended, while the usual roundrobin case degenerates to an impure form of fifo scheduling (iit is impure since priority decay still works so it is only fifo among threads of the same priority). >>... >>> @@ -2144,9 +2153,6 @@ >>> =09=09if >>> (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) >>> tdq->tdq_ridx =3D tdq->tdq_idx; } >>> -=09ts =3D td->td_sched; >>> -=09if (td->td_pri_class & PRI_FIFO_BIT) >>> -=09=09return; >>> =09if (PRI_BASE(td->td_pri_class) =3D=3D PRI_TIMESHARE) { >>> =09=09/* >>> =09=09 * We used a tick; charge it to the thread so >>> @@ -2157,11 +2163,6 @@ >>> =09=09sched_priority(td); >>> =09} >>> =09/* >>> -=09 * We used up one time slice. >>> -=09 */ >>> -=09if (--ts->ts_slice > 0) >>> -=09=09return; >>> -=09/* >>> =09 * We're out of time, force a requeue at userret(). >>> =09 */ >>> =09ts->ts_slice =3D sched_slice; With the ts_slice check here before you moved it, removing it might give buggy behaviour closer to SCHED_4BSD. >>> and refusal to use options FULL_PREEMPTION 4-5 years ago, I found that any form of PREMPTION was a pessimization for at least makeworld (since it caused too many context switches). PREEMPTION was needed for the !SMP case, at least partly because of the broken switchticks (switchticks, when it works, gives voluntary yielding by some CPU hogs in the kernel. PREEMPTION, if it works, should do this better). So I used PREEMPTION in the !SMP case and not for the SMP case. I didn't worry about the CPU hogs in the SMP case since it is rare to have more than 1 of them and 1 will use at most 1/2 of a multi-CPU system. >>> But no one has unsubscribed to my letter, my patch helps or not in >>> the case of Core2Duo... >>> There is a suspicion that the problems stem from the sections of >>> code associated with the SMP... >>> Maybe I'm in something wrong, but I want to help in solving this >>> problem ... The main point of SCHED_ULE is to give better affinity for multi-CPU systems. But the `multi' apparently needs to be strictly more than 2 for it to brak even. Bruce --0-1963381402-1323825914=:1563-- From owner-freebsd-performance@FreeBSD.ORG Wed Dec 14 16:59:53 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D732A1065676; Wed, 14 Dec 2011 16:59:53 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 919838FC19; Wed, 14 Dec 2011 16:59:53 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBEGxlPE000375; Wed, 14 Dec 2011 11:59:47 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EE8D607.1000504@sentex.net> Date: Wed, 14 Dec 2011 11:59:51 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: mdf@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Ivan Klymenko , Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 16:59:54 -0000 On 12/13/2011 7:01 PM, mdf@freebsd.org wrote: > > Has anyone experiencing problems tried to set sysctl kern.sched.steal_thresh=1 ? > > I don't remember what our specific problem at $WORK was, perhaps it > was just interrupt threads not getting serviced fast enough, but we've > hard-coded this to 1 and removed the code that sets it in > sched_initticks(). The same effect should be had by setting the > sysctl after a box is up. FWIW, this does impact the performance of pbzip2 on an i7. Using a 1.1G file pbzip2 -v -c big > /dev/null with burnP6 running in the background, sysctl kern.sched.steal_thresh=1 vs sysctl kern.sched.steal_thresh=3 N Min Max Median Avg Stddev x 10 38.005022 38.42238 38.194648 38.165052 0.15546188 + 9 38.695417 40.595544 39.392127 39.435384 0.59814114 Difference at 95.0% confidence 1.27033 +/- 0.412636 3.32852% +/- 1.08119% (Student's t, pooled s = 0.425627) a value of 1 is *slightly* faster. -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-performance@FreeBSD.ORG Wed Dec 14 17:55:53 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34EE01065679; Wed, 14 Dec 2011 17:55:53 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id D3CA08FC17; Wed, 14 Dec 2011 17:55:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=H5HuC6f1zLZn/iZE9rSzVBa6GimuIdr5WV8rZoyf9bw=; b=P0a5+C+QspdhOkFWZHN5MJx+rooqxJQlNCwltxzVAO88ay7bFsjjzeY2MqGt/+9YIJ/mC0vkLkMoFTbNcYNuJ3bKdvQo7vwgj/Xbngp+M2usczXPalskGQ2vzlbYNGVUVyptOud0DluvzMXe3sV+ttAaRd1sR8dQCWPYAdOAQCA=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1Rat3X-000O37-NU ; Wed, 14 Dec 2011 19:55:51 +0200 Date: Wed, 14 Dec 2011 19:55:49 +0200 From: Ivan Klymenko To: Andrey Chernov Message-ID: <20111214195549.415196f0@nonamehost.> In-Reply-To: <20111214173435.GA41893@vniz.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213090051.GA3339@vniz.net> <20111214173435.GA41893@vniz.net> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Adrian Chadd , Doug Barton , freebsd-stable@FreeBSD.ORG, "O. Hartmann" , Current FreeBSD , freebsd-performance@FreeBSD.ORG Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 17:55:53 -0000 =D0=92 Wed, 14 Dec 2011 21:34:35 +0400 Andrey Chernov =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > On Tue, Dec 13, 2011 at 02:22:48AM -0800, Adrian Chadd wrote: > > On 13 December 2011 01:00, Andrey Chernov wrote: > >=20 > > >> If the algorithm ULE does not contain problems - it means the > > >> problem has Core2Duo, or in a piece of code that uses the ULE > > >> scheduler. > > > > > > I observe ULE interactivity slowness even on single core machine > > > (Pentium 4) in very visible places, like 'ps ax' output stucks in > > > the middle by ~1 second. When I switch back to SHED_4BSD, all > > > slowness is gone. > >=20 > > Are you able to provide KTR traces of the scheduler results? > > Something that can be fed to schedgraph? >=20 > Sorry, this machine is not mine anymore. I try SCHED_ULE on Core 2 > Duo instead and don't notice this effect, but it is overall pretty > fast comparing to that Pentium 4. >=20 Give me, please, detailed instructions on how to do it - I'll do it ... Be a shame if this the theme is will end again just only the discussions ... :( From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 21:58:23 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA1D9106566B; Tue, 13 Dec 2011 21:58:23 +0000 (UTC) (envelope-from malin.randstrom@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5CF768FC12; Tue, 13 Dec 2011 21:58:23 +0000 (UTC) Received: by ggnp1 with SMTP id p1so178082ggn.13 for ; Tue, 13 Dec 2011 13:58:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=sNUT/fjw6EchSejWXjJP78ubljFlvCF9/iUH/FOJoMw=; b=PfIJj08Hg3htPuqeML1U2v1eJXBM77qQyqpdPPq+VNEjSULs4zPKRI2Bu18wWL237o qEd1GrX4vdWLE4fWlX5YDR87QWLA8rFXDsoHM0bOJHPTeLEVpYw+5yK3zG3cQrDSn+OU AfZ94IGdq58983Kbm6A1Y3ZAN9v/1UPA4MGOw= MIME-Version: 1.0 Received: by 10.182.218.100 with SMTP id pf4mr4801602obc.12.1323811881013; Tue, 13 Dec 2011 13:31:21 -0800 (PST) Received: by 10.182.6.133 with HTTP; Tue, 13 Dec 2011 13:31:20 -0800 (PST) Received: by 10.182.6.133 with HTTP; Tue, 13 Dec 2011 13:31:20 -0800 (PST) In-Reply-To: <20111213155456.GA93017@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE751E2.60204@mail.zedat.fu-berlin.de> <20111213155456.GA93017@troutmask.apl.washington.edu> Date: Tue, 13 Dec 2011 22:31:20 +0100 Message-ID: From: Malin Randstrom To: Steve Kargl X-Mailman-Approved-At: Wed, 14 Dec 2011 18:00:27 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Current FreeBSD , freebsd-stable@freebsd.org, "O. Hartmann" Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: malin@randstrom.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 21:58:23 -0000 stop sending me spam mail ... you never stop despite me having unsubscribeb several times. stop this! On Dec 13, 2011 8:12 PM, "Steve Kargl" wrote: > On Tue, Dec 13, 2011 at 02:23:46PM +0100, O. Hartmann wrote: > > On 12/12/11 16:51, Steve Kargl wrote: > > > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > > >> > > >>> Not fully right, boinc defaults to run on idprio 31 so this isn't an > > >>> issue. And yes, there are cases where SCHED_ULE shows much better > > >>> performance then SCHED_4BSD. [...] > > >> > > >> Do we have any proof at hand for such cases where SCHED_ULE performs > > >> much better than SCHED_4BSD? Whenever the subject comes up, it is > > >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > > > > >> 2. But in the end I see here contradictionary statements. People > > >> complain about poor performance (especially in scientific > environments), > > >> and other give contra not being the case. > > >> > > >> Within our department, we developed a highly scalable code for > planetary > > >> science purposes on imagery. It utilizes present GPUs via OpenCL if > > >> present. Otherwise it grabs as many cores as it can. > > >> By the end of this year I'll get a new desktop box based on Intels new > > >> Sandy Bridge-E architecture with plenty of memory. If the colleague > who > > >> developed the code is willing performing some benchmarks on the same > > >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most > > >> recent Suse. For FreeBSD I intent also to look for performance with > both > > >> different schedulers available. > > >> > > > > > > This comes up every 9 months or so, and must be approaching > > > FAQ status. > > > > > > In a HPC environment, I recommend 4BSD. Depending on > > > the workload, ULE can cause a severe increase in turn > > > around time when doing already long computations. If > > > you have an MPI application, simply launching greater > > > than ncpu+1 jobs can show the problem. > > > > Well, those recommendations should based on "WHY". As the mostly > > negative experiences with SCHED_ULE in highly computative workloads get > > allways contradicted by "...but there are workloads that show the > > opposite ..." this should be shown by more recent benchmarks and > > explanations than legacy benchmarks from years ago. > > > > I have given the WHY in previous discussions of ULE, based > on what you call legacy benchmarks. I have not seen any > commit to sched_ule.c that would lead me to believe that > the performance issues with ULE and cpu-bound numerical > codes have been addressed. Repeating the benchmark would > be a waste of time. > > -- > Steve > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 22:03:51 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 95AED1065679; Tue, 13 Dec 2011 22:03:51 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from 172-17-198-245.globalsuite.net (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 391AD162704; Tue, 13 Dec 2011 22:03:51 +0000 (UTC) Message-ID: <4EE7CBC6.5010001@FreeBSD.org> Date: Tue, 13 Dec 2011 14:03:50 -0800 From: Doug Barton Organization: http://SupersetSolutions.com/ User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111110 Thunderbird/8.0 MIME-Version: 1.0 To: malin@randstrom.com References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111212155159.GB73597@troutmask.apl.washington.edu> <4EE751E2.60204@mail.zedat.fu-berlin.de> <20111213155456.GA93017@troutmask.apl.washington.edu> In-Reply-To: X-Enigmail-Version: undefined OpenPGP: id=1A1ABC84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 14 Dec 2011 18:00:27 +0000 Cc: freebsd-stable@freebsd.org, Malin Randstrom , "O. Hartmann" , Current FreeBSD , Steve Kargl , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 22:03:51 -0000 On 12/13/2011 13:31, Malin Randstrom wrote: > stop sending me spam mail ... you never stop despite me having unsubscribeb > several times. stop this! If you had actually unsubscribed, the mail would have stopped. :) You can see the instructions you need to follow below. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > -- [^L] Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 23:04:44 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83997106564A; Tue, 13 Dec 2011 23:04:43 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) by mx1.freebsd.org (Postfix) with ESMTP id 19D0D8FC17; Tue, 13 Dec 2011 23:04:43 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 4714C1DD4FB; Wed, 14 Dec 2011 00:04:42 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 1F56528468; Wed, 14 Dec 2011 00:04:42 +0100 (CET) Date: Wed, 14 Dec 2011 00:04:42 +0100 From: Jilles Tjoelker To: Ivan Klymenko Message-ID: <20111213230441.GB42285@stack.nl> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111213104048.40f3e3de@nonamehost.> User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Wed, 14 Dec 2011 18:00:27 +0000 Cc: "O. Hartmann" , Doug Barton , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Current FreeBSD Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 23:04:44 -0000 On Tue, Dec 13, 2011 at 10:40:48AM +0200, Ivan Klymenko wrote: > If the algorithm ULE does not contain problems - it means the problem > has Core2Duo, or in a piece of code that uses the ULE scheduler. > I already wrote in a mailing list that specifically in my case (Core2Duo) > partially helps the following patch: > --- sched_ule.c.orig 2011-11-24 18:11:48.000000000 +0200 > +++ sched_ule.c 2011-12-10 22:47:08.000000000 +0200 > @@ -794,7 +794,8 @@ > * 1.5 * balance_interval. > */ > balance_ticks = max(balance_interval / 2, 1); > - balance_ticks += random() % balance_interval; > +// balance_ticks += random() % balance_interval; > + balance_ticks += ((int)random()) % balance_interval; > if (smp_started == 0 || rebalance == 0) > return; > tdq = TDQ_SELF(); This avoids a 64-bit division on 64-bit platforms but seems to have no effect otherwise. Because this function is not called very often, the change seems unlikely to help. > @@ -2118,13 +2119,21 @@ > struct td_sched *ts; > > THREAD_LOCK_ASSERT(td, MA_OWNED); > + if (td->td_pri_class & PRI_FIFO_BIT) > + return; > + ts = td->td_sched; > + /* > + * We used up one time slice. > + */ > + if (--ts->ts_slice > 0) > + return; This skips most of the periodic functionality (long term load balancer, saving switch count (?), insert index (?), interactivity score update for long running thread) if the thread is not going to be rescheduled right now. It looks wrong but it is a data point if it helps your workload. > tdq = TDQ_SELF(); > #ifdef SMP > /* > * We run the long term load balancer infrequently on the first cpu. > */ > - if (balance_tdq == tdq) { > - if (balance_ticks && --balance_ticks == 0) > + if (balance_ticks && --balance_ticks == 0) { > + if (balance_tdq == tdq) > sched_balance(); > } > #endif The main effect of this appears to be to disable the long term load balancer completely after some time. At some point, a CPU other than the first CPU (which uses balance_tdq) will set balance_ticks = 0, and sched_balance() will never be called again. It also introduces a hypothetical race condition because the access to balance_ticks is no longer restricted to one CPU under a spinlock. If the long term load balancer may be causing trouble, try setting kern.sched.balance_interval to a higher value with unpatched code. > @@ -2144,9 +2153,6 @@ > if (TAILQ_EMPTY(&tdq->tdq_timeshare.rq_queues[tdq->tdq_ridx])) > tdq->tdq_ridx = tdq->tdq_idx; > } > - ts = td->td_sched; > - if (td->td_pri_class & PRI_FIFO_BIT) > - return; > if (PRI_BASE(td->td_pri_class) == PRI_TIMESHARE) { > /* > * We used a tick; charge it to the thread so > @@ -2157,11 +2163,6 @@ > sched_priority(td); > } > /* > - * We used up one time slice. > - */ > - if (--ts->ts_slice > 0) > - return; > - /* > * We're out of time, force a requeue at userret(). > */ > ts->ts_slice = sched_slice; > and refusal to use options FULL_PREEMPTION > But no one has unsubscribed to my letter, my patch helps or not in the > case of Core2Duo... > There is a suspicion that the problems stem from the sections of code > associated with the SMP... > Maybe I'm in something wrong, but I want to help in solving this > problem ... -- Jilles Tjoelker From owner-freebsd-performance@FreeBSD.ORG Tue Dec 13 23:19:40 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34672106567A; Tue, 13 Dec 2011 23:19:40 +0000 (UTC) (envelope-from marcus@odin.blazingdot.com) Received: from odin.blazingdot.com (odin.blazingdot.com [199.48.133.254]) by mx1.freebsd.org (Postfix) with ESMTP id 1347F8FC16; Tue, 13 Dec 2011 23:19:39 +0000 (UTC) Received: by odin.blazingdot.com (Postfix, from userid 1001) id 01B5D114241; Tue, 13 Dec 2011 23:02:15 +0000 (UTC) Date: Tue, 13 Dec 2011 23:02:15 +0000 From: Marcus Reid To: Doug Barton Message-ID: <20111213230215.GA83159@blazingdot.com> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EE69C5A.3090005@FreeBSD.org> X-Coffee-Level: nearly-fatal User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Wed, 14 Dec 2011 18:00:27 +0000 Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Dec 2011 23:19:40 -0000 On Mon, Dec 12, 2011 at 04:29:14PM -0800, Doug Barton wrote: > On 12/12/2011 05:47, O. Hartmann wrote: > > Do we have any proof at hand for such cases where SCHED_ULE performs > > much better than SCHED_4BSD? > > I complained about poor interactive performance of ULE in a desktop > environment for years. I had numerous people try to help, including > Jeff, with various tunables, dtrace'ing, etc. The cause of the problem > was never found. The issues that I've seen with ULE on the desktop seem to be caused by X taking up a steady amount of CPU, and being demoted from being an "interactive" process. X then becomes the bottleneck for other processes that would otherwise be "interactive". Try 'renice -20 ' and see if that makes your problems go away. Marcus From owner-freebsd-performance@FreeBSD.ORG Wed Dec 14 17:34:38 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8475E106564A; Wed, 14 Dec 2011 17:34:38 +0000 (UTC) (envelope-from ache@vniz.net) Received: from vniz.net (vniz.net [194.87.13.69]) by mx1.freebsd.org (Postfix) with ESMTP id EC28B8FC0C; Wed, 14 Dec 2011 17:34:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by vniz.net (8.14.5/8.14.5) with ESMTP id pBEHYaPU041992; Wed, 14 Dec 2011 21:34:36 +0400 (MSK) (envelope-from ache@vniz.net) Received: (from ache@localhost) by localhost (8.14.5/8.14.5/Submit) id pBEHYZP4041991; Wed, 14 Dec 2011 21:34:35 +0400 (MSK) (envelope-from ache) Date: Wed, 14 Dec 2011 21:34:35 +0400 From: Andrey Chernov To: Adrian Chadd Message-ID: <20111214173435.GA41893@vniz.net> Mail-Followup-To: Andrey Chernov , Adrian Chadd , Ivan Klymenko , Doug Barton , "O. Hartmann" , Current FreeBSD , freebsd-stable@FreeBSD.ORG, freebsd-performance@FreeBSD.ORG References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213090051.GA3339@vniz.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Wed, 14 Dec 2011 18:00:27 +0000 Cc: Ivan Klymenko , Doug Barton , freebsd-stable@FreeBSD.ORG, "O. Hartmann" , Current FreeBSD , freebsd-performance@FreeBSD.ORG Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Dec 2011 17:34:38 -0000 On Tue, Dec 13, 2011 at 02:22:48AM -0800, Adrian Chadd wrote: > On 13 December 2011 01:00, Andrey Chernov wrote: > > >> If the algorithm ULE does not contain problems - it means the problem > >> has Core2Duo, or in a piece of code that uses the ULE scheduler. > > > > I observe ULE interactivity slowness even on single core machine (Pentium > > 4) in very visible places, like 'ps ax' output stucks in the middle by ~1 > > second. When I switch back to SHED_4BSD, all slowness is gone. > > Are you able to provide KTR traces of the scheduler results? Something > that can be fed to schedgraph? Sorry, this machine is not mine anymore. I try SCHED_ULE on Core 2 Duo instead and don't notice this effect, but it is overall pretty fast comparing to that Pentium 4. -- http://ache.vniz.net/ From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 00:02:05 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 788691065677; Thu, 15 Dec 2011 00:02:05 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 330CF8FC1E; Thu, 15 Dec 2011 00:02:05 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1Raylw-0001S6-DD>; Thu, 15 Dec 2011 01:02:04 +0100 Received: from e178035148.adsl.alicedsl.de ([85.178.35.148] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1Raylw-0006CG-8O>; Thu, 15 Dec 2011 01:02:04 +0100 Message-ID: <4EE938FB.7010107@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 01:02:03 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Current FreeBSD , freebsd-performance@freebsd.org X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig10A02CAC22836E31F0341B77" X-Originating-IP: 85.178.35.148 Cc: Subject: NEWS: NVIDIA Open-Sources Its CUDA Compiler X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 00:02:05 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig10A02CAC22836E31F0341B77 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: quoted-printable Just read this on phoronix.com Is this finally a chance to get GPGPU on FreeBSD natively supported? nVidia has a binary driver, supporting well their higher end graphics cards on FreeBSD 64bit natively. I do not understand much about the compiler itself, it's "nvcc" as far as I know, and it is also doing well OpenCL (with some serious bugs we revealed). What would be needed to bring FreeBSd finally back to the HPC scenario with being capable of dealing natively with GPGPU stuff on nVidia graphics cards? There are libraries installed by the driver or the SDK. With a OpenSource compiler it should also be possible for nVidia, assumed the compiler works with freeBSD natively, to provide OpenCL stuff as well as CUDA stuff. Please correct me and destroy me "dreams" having FreeBSD in my lab working on GPUs ... The decission sounds like some pitfall in a contract. Is nVidia dropping CUDA in favour of OpenCL or is the CUDA compiler only a tiny piece of the whole thing that could be easily considered open source without changing the "great restricted Linux-only" picture? Maybe LLVM, now part of FreeBSD's backbone, is capable of taking advantage of the opening of the CUDA compiler so we will see a combination of CLANG/OpenCL/CUDA soon on FreeBSD introduced by LLVM? Well, well, this is awesome ... ;-) Oliver --------------enig10A02CAC22836E31F0341B77 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6Tj7AAoJEOgBcD7A/5N8iNUH/iJjA91F49ONtont656fLroX E2VMsTqxiHARb5Ef/tlkg3nQl3yQF8iHMn2qr15BNoRIIAtBY7SYnIcvOk6bPQSY tRTpMXA2YrAtQmanD+4IrHW3oypWJ9Ubu/96M0Kj8jSJ+F3rAwnyNblT50AsAQ2r qLomIhotSKL25qN91O6sPdigG3q+ZvhMwvjuVLIK/eGx8z6u4BZ/BOpDlDv0KBTZ KADxMBLuykdmrz3afyqAGhmuTGqh8evvT7oc6j/ZgxkxVxJjCPQZF/0xDpMI08p1 XaiMK05Qdww0Sf/QO24GlPtgtA7wxdx5/OEHX+bRsTtfzNRRSnc1j7OcU8mWFps= =v8J1 -----END PGP SIGNATURE----- --------------enig10A02CAC22836E31F0341B77-- From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 07:32:57 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FF881065672; Thu, 15 Dec 2011 07:32:57 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id F31158FC16; Thu, 15 Dec 2011 07:32:56 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1Rb5oE-0005nB-UC>; Thu, 15 Dec 2011 08:32:55 +0100 Received: from e178002216.adsl.alicedsl.de ([85.178.2.216] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1Rb5oE-0007kp-PT>; Thu, 15 Dec 2011 08:32:54 +0100 Message-ID: <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 08:32:48 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Jeremy Chadwick , freebsd-performance@freebsd.org, Current FreeBSD References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> In-Reply-To: <20111215024249.GA13557@icarus.home.lan> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig3AC99262226F5AAF3DFA6856" X-Originating-IP: 85.178.2.216 Cc: FreeBSD Stable Mailing List Subject: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 07:32:57 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig3AC99262226F5AAF3DFA6856 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Just saw this shot benchmark on Phoronix dot com today: http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTAyNzA It may be worth to discuss the sad performance of FBSD in some parts of the benchmark. A difference of a factor 10 or 100 is simply far beyond disapointing, it is more than inacceptable and by just reading those benchmarks, I'd like to drop thinking of using FreeBSD even as a backend server in scientific and business environments. In detail, some of the SciMark benches look disappointing. The overall image can't help over the fact that in C-Ray FreeBSD is better performing. =46rom the compiler, I'd like say there couldn't be a drop of more than 1= 0 - 15% in performance - but not 10 or 100 times. I'm just thinking about the discussion of SCHED_ULE and all the saur spots we discussed when I stumbled over the test. Regards, Oliver --------------enig3AC99262226F5AAF3DFA6856 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6aKmAAoJEOgBcD7A/5N8dvYH/1pfvvuy5BJWEEf5LTNhAaIv awHwt5jJ3WJA7zmwtnbGSw2qkRFC8E9D5+jOQ0rissGrSYH4qBakSpfnnJiRTtOm iYzAwnQYXt2STTKuNaz4rcG3bnX8i1SpbHre6Kj1p4cij/sQJXty9CMdVIR3dwYD pLfxSk9yFYrWi2Xpy9zqxdMKC1g/FITIuScwQeXtD3tfQlrh+LPvDK21c+OhukeZ cgVuzNw2274pTPlLNaJpAGkcMw1kPJ3U1cEGaI4nwGLFKvduQp2z13mRHLXATh/a lmVvV/0AIJ6UVLGpwOaBcaCXFxWJ+ez9aDlYM18z2dlfOvLnYcxND/u5GoHHyJg= =3PDg -----END PGP SIGNATURE----- --------------enig3AC99262226F5AAF3DFA6856-- From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 07:40:34 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 188321065672; Thu, 15 Dec 2011 07:40:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 89E108FC16; Thu, 15 Dec 2011 07:40:33 +0000 (UTC) Received: by vbbfr13 with SMTP id fr13so2103260vbb.13 for ; Wed, 14 Dec 2011 23:40:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=z/7NpF07+xvfN4qI0sYfDhw2Q31cy25YPhN7EvKe9tI=; b=xYzKxuzfXTM9YnmvW9fwiVbRDobNVzSTGbhecrQTSS+SdcqwJI9Aq97K6nmz45H3Ya Hja4UYIFjpodGDLXcoHbuDdt9JFShzlS1R7W7AG0Z6ln17JzHfIVNKKR38b+BgQToMoe uZqsruqX6c8W9+6gDbRzq7kLJEA/k6bnmECbE= MIME-Version: 1.0 Received: by 10.52.67.111 with SMTP id m15mr1625754vdt.96.1323934832733; Wed, 14 Dec 2011 23:40:32 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.26.50 with HTTP; Wed, 14 Dec 2011 23:40:32 -0800 (PST) In-Reply-To: <4EE9A2A0.80607@zedat.fu-berlin.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Wed, 14 Dec 2011 23:40:32 -0800 X-Google-Sender-Auth: URhgjaXgcqV2vgLQyrCg_OMuz-M Message-ID: From: Adrian Chadd To: "O. Hartmann" Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 07:40:34 -0000 On 14 December 2011 23:32, O. Hartmann wrote: > Just saw this shot benchmark on Phoronix dot com today: > > http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA > > It may be worth to discuss the sad performance of FBSD in some parts of > the benchmark. A difference of a factor 10 or 100 is simply far beyond Well, the only way it's going to get fixed is if someone sits down, replicates it, and starts to document exactly what it is that these benchmarks are/aren't doing. Sometimes it's because the benchmark is very much tickling things incorrectly. In a lot of cases though, the benchmark is testing something synthetic that Linux just happens to have micro-optimised. So if you care about this a lot, someone needs to stand up, work with Phronix to get some actual feedback about what's going on, and see if it can be fixed. Maybe you'll find ULE is broken in some instances; I bet you'll find something like "the disk driver is suboptimal." For example, I remember seeing someone mess up a test because they split their filesystems across raid5 boundaries, and this was hidden by the choice of raid controller and stripe size. This made FreeBSD look worse; when this was corrected for, it sped up far past Linux. Adrian From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 11:14:23 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA6801065672; Thu, 15 Dec 2011 11:14:23 +0000 (UTC) (envelope-from prvs=1330762df1=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id E89548FC13; Thu, 15 Dec 2011 11:14:22 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 11:02:29 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 15 Dec 2011 11:02:24 +0000 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50017140276.msg; Thu, 15 Dec 2011 11:02:23 +0000 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1330762df1=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <95A7C3EFE7B4406EAFF0259CD8B61D1C@multiplay.co.uk> From: "Steven Hartland" To: "Michael Larabel" , "Michael Ross" References: <4EE1EAFE.3070408@m5p.com><4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com><4EE933C6.4020209@zedat.fu-berlin.de><20111215024249.GA13557@icarus.home.lan><4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9D214.3070906@phoronix.com> Date: Thu, 15 Dec 2011 11:02:47 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-15"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 11:14:24 -0000 ----- Original Message ----- From: "Michael Larabel" > > I was the on that carried out the testing and know that it was on the > same system. > > All of the testing, including the system tables, is fully automated. > Under FreeBSD sometimes the parsing of some component strings isn't as > nice as Linux and other supported operating systems by the Phoronix Test > Suite. For the BSD motherboard string parsing it's grabbing > hw.vendor/hw.product from sysctl. Is there a better place to read the > motherboard DMI information from? dmidecode may provide better info? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 14:59:58 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32F81106566B for ; Thu, 15 Dec 2011 14:59:58 +0000 (UTC) (envelope-from pcallycat@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id B14638FC12 for ; Thu, 15 Dec 2011 14:59:57 +0000 (UTC) Received: by wgbdr11 with SMTP id dr11so4099632wgb.31 for ; Thu, 15 Dec 2011 06:59:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=sqavD36XWaPJZuM6zSWfgGVPNK6XwdDB/5Tnjve37pI=; b=sNH23W1HjiIPNtiNCBQc/oVoSYd5wVnMxmOzjZrvJbaJtMe+pHQcVJCeY2n71N6cUq HKO4SZJyRC5llEbMmMS3dt2v2h8ma98FatzzvbADivJUDLIMpOIkZRtDFdrmOLadLcnf nC5s08NV+9lwkEyEDjp9Jyi1whDf9icZ9bICk= Received: by 10.227.205.135 with SMTP id fq7mr2434587wbb.19.1323959560512; Thu, 15 Dec 2011 06:32:40 -0800 (PST) Received: from [192.168.1.100] (71-221-190-191.bois.qwest.net. [71.221.190.191]) by mx.google.com with ESMTPS id fk3sm9552157wbb.10.2011.12.15.06.32.37 (version=SSLv3 cipher=OTHER); Thu, 15 Dec 2011 06:32:38 -0800 (PST) Message-ID: <4EEA04FB.7060905@gmail.com> Date: Thu, 15 Dec 2011 07:32:27 -0700 From: Mike Bedwell User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> In-Reply-To: <4EE9A2A0.80607@zedat.fu-berlin.de> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 14:59:58 -0000 On 12/15/2011 12:32 AM, O. Hartmann wrote: > Just saw this shot benchmark on Phoronix dot com today: > > http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA > > It may be worth to discuss the sad performance of FBSD in some parts of > the benchmark. A difference of a factor 10 or 100 is simply far beyond > disapointing, it is more than inacceptable and by just reading those > benchmarks, I'd like to drop thinking of using FreeBSD even as a backend > server in scientific and business environments. In detail, some of the > SciMark benches look disappointing. The overall image can't help over > the fact that in C-Ray FreeBSD is better performing. > > From the compiler, I'd like say there couldn't be a drop of more than 10 > - 15% in performance - but not 10 or 100 times. > > I'm just thinking about the discussion of SCHED_ULE and all the saur > spots we discussed when I stumbled over the test. > > Regards, > Oliver > The benchmarks also need to be ran on equivalent hardware. I've yet to see phoronix perform a benchmark test where they actually used the same rig to perform benchmark comparisons. Too many things change from one benchmark to the next to be able to reliably say what is at fault for the benchmark differences. The test needs to be re-ran in an environment where the only thing that changes, is the operating system. These benchmarks only show that linux on one machine performs differently than bsd on an entirely different machine. From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 15:01:50 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2DC86106566B for ; Thu, 15 Dec 2011 15:01:50 +0000 (UTC) (envelope-from afmcc@btinternet.com) Received: from nm3.bt.bullet.mail.ird.yahoo.com (nm3.bt.bullet.mail.ird.yahoo.com [212.82.108.234]) by mx1.freebsd.org (Postfix) with SMTP id 8CED28FC18 for ; Thu, 15 Dec 2011 15:01:49 +0000 (UTC) Received: from [212.82.108.230] by nm3.bt.bullet.mail.ird.yahoo.com with NNFMP; 15 Dec 2011 14:49:30 -0000 Received: from [212.82.108.226] by tm3.bt.bullet.mail.ird.yahoo.com with NNFMP; 15 Dec 2011 14:49:30 -0000 Received: from [127.0.0.1] by omp1003.bt.mail.ird.yahoo.com with NNFMP; 15 Dec 2011 14:49:30 -0000 X-Yahoo-Newman-Id: 46271.4245.bm@omp1003.bt.mail.ird.yahoo.com Received: (qmail 24702 invoked from network); 15 Dec 2011 14:49:29 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=btinternet.com; h=DKIM-Signature:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:Date:From:To:Subject:Message-ID:In-Reply-To:References:X-Mailer:Mime-Version:Content-Type:Content-Transfer-Encoding; b=06eHJP+zd/LUhAKdcKNoMIMoWr/99Nw4C4Z9jPR/uWBo4+XMxdZb0FpvtBwOxY1bY8k+D3AYjtWlGmEfk2eVglzX03aagjq4HmubjHETTLtT2PRGg39efVYkSr/VOwHubYD5UqvZeTYSqaICphEAchwECXBF76rvbq37FjP9sgA= ; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=btinternet.com; s=s1024; t=1323960569; bh=2d+eLHedHIlvVskXP2K9EcsgPyFgU6e7O0GcANkhmio=; h=X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:Date:From:To:Subject:Message-ID:In-Reply-To:References:X-Mailer:Mime-Version:Content-Type:Content-Transfer-Encoding; b=h0xrbNVTQn4kDLf/BnC8lgxRPl743SBs2BVNFAqKStXT9BBFIP+cXv04yYSI/H5AszwMrZptU0q/YloR5+V/pnke1j57znvHX3NjVuM4YsdCicuF1CXHlp2vDB+8yF1cb6iFXgfc0LCoN1AcXGLN6UfzYIHUgptquKBkH9ShK9M= X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: YvpKvJAVM1nawniYxIUA0.b0.oL17actkKS.xsKjdzW0Ig3 _ibuFZ4gvQINMk.TyhuemUN_AetYs96GgESB_olzOEOFM3T8F5b2ReWsuFuL 6U3wrCh7mR6jtL9jZhf9bt375RVoofpyPYgXx8IG3S22E5XZLFR7nv2CYuXk ChPGHeCoYLzWEXmMiD3ZLLiivbnX502A8hXeQyy8as2961TsgjRlAUeL__XL mV4VgcUbiuw4T_yde2AR19vQDTZPbE2xh8ffmItujKOHz1rv0nk.wR57Qg9L 9RnAjgTFUEJ3ZAYqjhY_ShJYbgGWEeigLHGxuwazj7Qa3q3wH5NQr4LqcpuD 3SPvofc0AzcTpHxSbLy0iF7w0ebsd0X66.LIsEcadvU.GDI.w2VNq.rPQ1MU LMe7oCCjWQOWo5SigVA5okpmjIqjaH_gRdBXCjKpEDg1qf1ByD_kfULFOvvu WKwTKhpg7TpIRV6ddFV2U_h5gouyCGbt5kohT2gehgrd_OYGb78KPNhfqr.l XhuqVK9gNpgIxRUeMjDKs.IykcnNO X-Yahoo-SMTP: GbC5zv6swBDAJAX2wjERvjXPaCXFiJJLdMa.NuzRNApZ Received: from elena.home (afmcc@86.167.97.113 with login) by smtp821.mail.ukl.yahoo.com with SMTP; 15 Dec 2011 14:49:24 +0000 GMT Date: Thu, 15 Dec 2011 14:49:24 +0000 From: Tony McC To: freebsd-performance@freebsd.org Message-ID: <20111215144924.61a08733@elena.home> In-Reply-To: <4EE9A2A0.80607@zedat.fu-berlin.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd9.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 15:01:50 -0000 On Thu, 15 Dec 2011 08:32:48 +0100 "O. Hartmann" wrote: > Just saw this shot benchmark on Phoronix dot com today: > > http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA > > It may be worth to discuss the sad performance of FBSD in some parts > of the benchmark. A difference of a factor 10 or 100 is simply far > beyond disapointing, it is more than inacceptable and by just reading > those benchmarks, I'd like to drop thinking of using FreeBSD even as > a backend server in scientific and business environments. In detail, > some of the SciMark benches look disappointing. The overall image > can't help over the fact that in C-Ray FreeBSD is better performing. > > From the compiler, I'd like say there couldn't be a drop of more than > 10 > - 15% in performance - but not 10 or 100 times. > > I'm just thinking about the discussion of SCHED_ULE and all the saur > spots we discussed when I stumbled over the test. > > Regards, > Oliver > I suggest always ignoring benchmarks. They are like reading the astrology column in a tabloid newspaper. Instead, try FreeBSD for your work. Is it fast enough? Surely that is all you need to know. FreeBSD is quite fast enough for my needs and I am simply more productive using it than when I use any other operating system. That is partly to do with my familiarity with my setup, which I have customised the way I want. That is something that no benchmark can allow for. Tony From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 15:48:40 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E78701065670; Thu, 15 Dec 2011 15:48:40 +0000 (UTC) (envelope-from pieter@degoeje.nl) Received: from smtp.utwente.nl (smtp1.utsp.utwente.nl [130.89.2.8]) by mx1.freebsd.org (Postfix) with ESMTP id 548BD8FC13; Thu, 15 Dec 2011 15:48:39 +0000 (UTC) Received: from [130.89.165.91] (nox.student.utwente.nl [130.89.165.91]) by smtp.utwente.nl (8.12.10/SuSE Linux 0.7) with ESMTP id pBFFmXD8023626; Thu, 15 Dec 2011 16:48:33 +0100 Message-ID: <4EEA16D1.2010900@degoeje.nl> Date: Thu, 15 Dec 2011 16:48:33 +0100 From: Pieter de Goeje User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.2) Gecko/20110902 Thunderbird/6.0.2 MIME-Version: 1.0 To: "O. Hartmann" References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> In-Reply-To: <4EE9A2A0.80607@zedat.fu-berlin.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-UTwente-MailScanner-Information: Scanned by MailScanner. Contact icts.servicedesk@utwente.nl for more information. X-UTwente-MailScanner: Found to be clean X-UTwente-MailScanner-From: pieter@degoeje.nl X-Spam-Status: No Cc: freebsd-performance@freebsd.org, Current FreeBSD Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 15:48:41 -0000 Op 15-12-2011 8:32, O. Hartmann schreef: > Just saw this shot benchmark on Phoronix dot com today: > > http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA > > It may be worth to discuss the sad performance of FBSD in some parts of > the benchmark. A difference of a factor 10 or 100 is simply far beyond > disapointing, it is more than inacceptable and by just reading those > benchmarks, I'd like to drop thinking of using FreeBSD even as a backend > server in scientific and business environments. In detail, some of the > SciMark benches look disappointing. The overall image can't help over > the fact that in C-Ray FreeBSD is better performing. > > From the compiler, I'd like say there couldn't be a drop of more than 10 > - 15% in performance - but not 10 or 100 times. > > I'm just thinking about the discussion of SCHED_ULE and all the saur > spots we discussed when I stumbled over the test. > > Regards, > Oliver Detailed results here: http://openbenchmarking.org/result/1112113-AR-ORACLELIN37 As usual, the phoronix benchmarks are very misleading. 1) The linked benchmarks were not run on the same hardware. Hardware is close but not completely equal; for instance different brands of disks were used. 2) They didn't use the same compiler. This is really bad and _can_ lead to more than a factor 2 performance difference. Especially in "scientific" programs where (auto) vectorization is very important. Why on earth the benchmarker was too lazy to install a more recent GCC I have no idea. Of all the benchmarks shown only the disk benchmarks are interesting, because they actually stress the system. Unfortunately they screwed that up too because they were performed on ZFS instead of the default, plain UFS which is a lot more like EXT4 in terms of functionality. The rest are pure CPU bound userspace workloads and I bet that if they were performed using the same compiler, similar results would've been achieved (barring any major VM differences). In any case we would've been able to actually compare FreeBSD vs Oracle Linux instead of GCC 4.5 vs 4.2. Now they are useless. I'm sorry if this mail sounds a bit harsh but I'm tired of seeing phoronix making the same elementary mistakes again and again even after these have been pointed out years ago. - Pieter From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 16:38:28 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2FD39106567C; Thu, 15 Dec 2011 16:38:28 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id DB9AC8FC15; Thu, 15 Dec 2011 16:38:27 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBFGcLBQ067563; Thu, 15 Dec 2011 11:38:21 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EEA227E.7080704@sentex.net> Date: Thu, 15 Dec 2011 11:38:22 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Attilio Rao References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:38:28 -0000 On 12/15/2011 11:26 AM, Attilio Rao wrote: > > Hi Mike, > was that just the same codebase with the switch SCHED_4BSD/SCHED_ULE? Hi Attilio, It was the same codebase. > Could you retry the bench checking CPU usage and possible thread > migration around for both cases? I can, but how do I do that ? ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 16:42:12 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E911106566C; Thu, 15 Dec 2011 16:42:12 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1CB118FC0A; Thu, 15 Dec 2011 16:42:10 +0000 (UTC) Received: by faaf16 with SMTP id f16so3582013faa.13 for ; Thu, 15 Dec 2011 08:42:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=8kUH6rJcxKtm25qBqXuAeBlmVWtVD5+Q5ma5HlD4/UY=; b=Ya4r1MsNOoyFFjTbRWnuqq4Ra5KhzIU8HozC5H8ltzEm+SBP4X8PZACbHOrS4yHl4Y 25bymJ4J8rp0Fqrsc9h4fzC4QOSmVi2D9lzjE9cfBRDhF+ZDj0bjcU078v8Clp34msjX 4gk07oPtDNL7ZAREgzlbPvlzDV+8w1/Ty9u+o= MIME-Version: 1.0 Received: by 10.180.96.103 with SMTP id dr7mr6732293wib.16.1323967329953; Thu, 15 Dec 2011 08:42:09 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:42:09 -0800 (PST) In-Reply-To: <4EEA227E.7080704@sentex.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> Date: Thu, 15 Dec 2011 17:42:09 +0100 X-Google-Sender-Auth: rWvfkJJt4u5T0-KL-sef3vLPTr0 Message-ID: From: Attilio Rao To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:42:12 -0000 2011/12/15 Mike Tancsa : > On 12/15/2011 11:26 AM, Attilio Rao wrote: >> >> Hi Mike, >> was that just the same codebase with the switch SCHED_4BSD/SCHED_ULE? > > Hi Attilio, > =C2=A0 =C2=A0 =C2=A0 =C2=A0It was the same codebase. > > >> Could you retry the bench checking CPU usage and possible thread >> migration around for both cases? > > I can, but how do I do that ? I'm thinking now to a better test-case for this: can you try that on a tmpfs volume? Also what filesystem you were using? How many CPUs were in place? Did you reboot before to move the steal_thresh value? Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 16:49:31 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03E101065675; Thu, 15 Dec 2011 16:49:31 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 61E528FC1A; Thu, 15 Dec 2011 16:49:30 +0000 (UTC) Received: by eekc50 with SMTP id c50so2711875eek.13 for ; Thu, 15 Dec 2011 08:49:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=qn2SX6Icz3xoH1/LZzTteZ2wkKvJHUss5vkQbFTGw88=; b=fgLJMqgdAh/wrr7kqbGLvkkEYs/vl6PwMjS1eUHzw+Eqwmeba6B1V4Bqvz3ZzFiXbA uBH1/bfrSGDC5L+hehANmNWfGcPqfSl4swHJjcmGBNxNONeZcJEIg1zVYOm3Hw0io+gZ U+OFVnzlUpdDILUKmKplz5cJrlJthd9/6/aY8= MIME-Version: 1.0 Received: by 10.180.74.211 with SMTP id w19mr6552377wiv.7.1323966364698; Thu, 15 Dec 2011 08:26:04 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:26:04 -0800 (PST) In-Reply-To: <4EE8D607.1000504@sentex.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> Date: Thu, 15 Dec 2011 17:26:04 +0100 X-Google-Sender-Auth: RQHg7a4l9VdsnVFWmvoxYxji0G8 Message-ID: From: Attilio Rao To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:49:31 -0000 2011/12/14 Mike Tancsa : > On 12/13/2011 7:01 PM, mdf@freebsd.org wrote: >> >> Has anyone experiencing problems tried to set sysctl kern.sched.steal_th= resh=3D1 ? >> >> I don't remember what our specific problem at $WORK was, perhaps it >> was just interrupt threads not getting serviced fast enough, but we've >> hard-coded this to 1 and removed the code that sets it in >> sched_initticks(). =C2=A0The same effect should be had by setting the >> sysctl after a box is up. > > FWIW, this does impact the performance of pbzip2 on an i7. Using a 1.1G f= ile > > pbzip2 -v -c big > /dev/null > > with burnP6 running in the background, > > sysctl kern.sched.steal_thresh=3D1 > vs > sysctl kern.sched.steal_thresh=3D3 > > > > =C2=A0 =C2=A0N =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Min =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Max =C2=A0 =C2=A0 =C2=A0 =C2=A0Median =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 Avg =C2=A0 =C2=A0 =C2=A0 =C2=A0Stddev > x =C2=A010 =C2=A0 =C2=A0 38.005022 =C2=A0 =C2=A0 =C2=A038.42238 =C2=A0 = =C2=A0 38.194648 =C2=A0 =C2=A0 38.165052 =C2=A0 =C2=A00.15546188 > + =C2=A0 9 =C2=A0 =C2=A0 38.695417 =C2=A0 =C2=A0 40.595544 =C2=A0 =C2=A0 = 39.392127 =C2=A0 =C2=A0 39.435384 =C2=A0 =C2=A00.59814114 > Difference at 95.0% confidence > =C2=A0 =C2=A0 =C2=A0 =C2=A01.27033 +/- 0.412636 > =C2=A0 =C2=A0 =C2=A0 =C2=A03.32852% +/- 1.08119% > =C2=A0 =C2=A0 =C2=A0 =C2=A0(Student's t, pooled s =3D 0.425627) > > a value of 1 is *slightly* faster. Hi Mike, was that just the same codebase with the switch SCHED_4BSD/SCHED_ULE? Also, the results here should be in the 3% interval for the avg case, which is not yet at the 'alarm level' but could still be an indication. I still suspect I/O plays a big role here, however, thus it could be detemined by other factors. Could you retry the bench checking CPU usage and possible thread migration around for both cases? Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 16:52:12 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 925D81065678; Thu, 15 Dec 2011 16:52:12 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 492BF8FC21; Thu, 15 Dec 2011 16:52:12 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBFGq9M5070372; Thu, 15 Dec 2011 11:52:09 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EEA25BB.7040706@sentex.net> Date: Thu, 15 Dec 2011 11:52:11 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Attilio Rao References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:52:12 -0000 On 12/15/2011 11:42 AM, Attilio Rao wrote: > > I'm thinking now to a better test-case for this: can you try that on a > tmpfs volume? There is enough RAM in the box so that it should not touch the disk, and I was sending the output to /dev/null, so it was not writing to the disk. > > Also what filesystem you were using? UFS > How many CPUs were in place? 4 > Did you reboot before to move the steal_thresh value? No. ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 16:54:46 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A57621065675 for ; Thu, 15 Dec 2011 16:54:46 +0000 (UTC) (envelope-from schulra@earlham.edu) Received: from chkenon.earlham.edu (chkenon.earlham.edu [159.28.1.87]) by mx1.freebsd.org (Postfix) with ESMTP id 4E2B68FC26 for ; Thu, 15 Dec 2011 16:54:45 +0000 (UTC) X-ASG-Debug-ID: 1323967030-037b9f16aa150e90001-3XdwJY Received: from tdream.lly.earlham.edu (tdream.lly.earlham.edu [159.28.7.241]) by chkenon.earlham.edu with ESMTP id C4amwHprtB3nCyz5; Thu, 15 Dec 2011 11:37:10 -0500 (EST) X-Barracuda-Envelope-From: schulra@earlham.edu X-Barracuda-Apparent-Source-IP: 159.28.7.241 Date: Thu, 15 Dec 2011 11:38:20 -0500 (EST) From: Randy Schultz X-X-Sender: schulra@tdream.lly.earlham.edu To: Pieter de Goeje In-Reply-To: <4EEA16D1.2010900@degoeje.nl> X-ASG-Orig-Subj: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server Message-ID: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EEA16D1.2010900@degoeje.nl> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Barracuda-Connect: tdream.lly.earlham.edu[159.28.7.241] X-Barracuda-Start-Time: 1323967030 X-Barracuda-URL: http://159.28.1.87:8000/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at earlham.edu X-Barracuda-Spam-Score: 0.89 X-Barracuda-Spam-Status: No, SCORE=0.89 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=1000.0 tests=BSF_SC5_SA210e, SARE_ADLTSUB4 X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.83162 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.89 SARE_ADLTSUB4 Apparent spam seems to contain porn subject 0.00 BSF_SC5_SA210e Custom Rule SA210e Cc: freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:54:46 -0000 On Thu, 15 Dec 2011, Pieter de Goeje spaketh thusly: -}Detailed results here: -}http://openbenchmarking.org/result/1112113-AR-ORACLELIN37 LOL! Pretty much 2 entirely different systems, even running different screen resolutions. Tnx for this link. -} -}As usual, the phoronix benchmarks are very misleading. Also, they tested fbsd RC2. This same thing has come up repeatedly. Seems to me "big waves" happened when fbsd 8.0 was coming out and phoronix tested RC1 or RC2. Unless my memory is in error (and it may well be), on the 8.0 "comparison" fiasco, it was pointed out that testing a fbsd RC release is like racing but being preventing from going full throttle. There are debugging hooks and various extra code bits that slow things down and are not taken out until the stable release. They *can* be taken out by the end-SA, but phoronix stated they used a stock kernel. That phoronix did this again makes me wonder... I have to agree with and cannot stress enough the importance of testing in the environment it is to be run in, with the software that is to be run on it. I used to be a massive linux fan, right up until the day I put freebsd up against several *nix boxen (IIRC Redhat, Debian, SuSE and IRIX) in a particular application I was re-working. I had to run the test several times, the difference was so great. Fbsd didn't just beat the others, it rolled 'em, smoked 'em and tapped them in the ashtray. But this was with _our_ hardware configurations and _our_ software configurations and tweaks. Currently we have a mixture of linux and fbsd in production and test. Some of the things we do run better on linux, some run better on fbsd. And if they're close, I'll pick fbsd mostly for personal reasons, e.g. it just makes more sense to me, some things I like to do are more easily done in fbsd, ... FWIW, YMMV, yadda yadda. ;> -- Randy (schulra@earlham.edu) 765.983.1283 <*> nosce te ipsum From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 16:56:38 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C11A106564A for ; Thu, 15 Dec 2011 16:56:38 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 304BD8FC1D for ; Thu, 15 Dec 2011 16:56:37 +0000 (UTC) Received: by eaaf13 with SMTP id f13so3011437eaa.13 for ; Thu, 15 Dec 2011 08:56:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=zm71dKbrHwDITPaDjF+CgkHwSvzGuXVS65Im4Up4UGA=; b=vy84MwsN3OjrK5jTpfX3KeozyU74p0n/QCFtpJkdxapqgaMyzMHrBEefLcpsxinmig qu6vaTOscLfWDyPT5APaxY5MNL5zcLqS7DN06TLqhWZAW9VtbR3AaVUpSL1+mloigV9E FRpDp64k/fatCR6282rafWlJbRDiY2eBPOvGI= MIME-Version: 1.0 Received: by 10.180.96.103 with SMTP id dr7mr6603082wib.16.1323966388055; Thu, 15 Dec 2011 08:26:28 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:26:27 -0800 (PST) In-Reply-To: <20111213073615.GA69641@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> Date: Thu, 15 Dec 2011 17:26:27 +0100 X-Google-Sender-Auth: 0ndvCp4x9Kuw_LOxQPoNVB24Q2Q Message-ID: From: Attilio Rao To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:56:38 -0000 2011/12/13 Jeremy Chadwick : > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: >> > Not fully right, boinc defaults to run on idprio 31 so this isn't an >> > issue. And yes, there are cases where SCHED_ULE shows much better >> > performance then SCHED_4BSD. =C2=A0[...] >> >> Do we have any proof at hand for such cases where SCHED_ULE performs >> much better than SCHED_4BSD? Whenever the subject comes up, it is >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu > >> 2. But in the end I see here contradictionary statements. People >> complain about poor performance (especially in scientific environments), >> and other give contra not being the case. >> >> Within our department, we developed a highly scalable code for planetary >> science purposes on imagery. It utilizes present GPUs via OpenCL if >> present. Otherwise it grabs as many cores as it can. >> By the end of this year I'll get a new desktop box based on Intels new >> Sandy Bridge-E architecture with plenty of memory. If the colleague who >> developed the code is willing performing some benchmarks on the same >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most >> recent Suse. For FreeBSD I intent also to look for performance with both >> different schedulers available. > > This is in no way shape or form the same kind of benchmark as what > you're planning to do, but I thought I'd throw it out there for folks to > take in as they see fit. > > I know folks were focused mainly on buildworld. > > I personally would find it interesting if someone with a higher-end > system (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the > same test (changing -jX to -j{numofcores} of course). > > -- > | Jeremy Chadwick =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0jdc at parodius.com= | > | Parodius Networking =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 http://www.parodius.com/ | > | UNIX Systems Administrator =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 Mountain View, CA, US | > | Making life hard for others since 1977. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 PGP 4BD6C0CB | > > > sched_ule > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > - time make -j2 buildworld > =C2=A01689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io 4565pf+0w > - time make -j2 buildkernel > =C2=A0640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w > > > sched_4bsd > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > - time make -j2 buildworld > =C2=A01662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io 6451pf+0= w > - time make -j2 buildkernel > =C2=A0638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w > > > software > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > * sched_ule test: =C2=A0FreeBSD 8.2-STABLE, Thu Dec =C2=A01 04:37:29 PST = 2011 > * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST 2011 Hi Jeremy, thanks for the time you spent on this. However, I wanted to ask/let you note 3 things: 1) Did you use 2 different code base for the test? (one updated on December 1 and another one on December 12) 2) Please note that you should have repeated this test several times (basically until you don't get a standard deviation which is acceptable with ministat) and report the ministat output 3) The difference is less than 2% which I suspect is really statistically unuseful/the same I'm not really even surprised ULE is not faster than 4BSD in this case because usually buildworld/buildkernel tests are driven for the vast majority by I/O overhead rather than scheduler capacity. It would be more interesting to analyze how buildworld does while another type of workload is going on. Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 16:56:40 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDDF31065689; Thu, 15 Dec 2011 16:56:40 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id A393F8FC13; Thu, 15 Dec 2011 16:56:39 +0000 (UTC) Received: by eaaf13 with SMTP id f13so3011475eaa.13 for ; Thu, 15 Dec 2011 08:56:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=Kek1F5J+opXNanowC4R0970JGtNQGCoNTh4rCJ7uQUs=; b=vwcrBfVBxoHdDZ0X9irxmOTnwQG2xiAdkINUqIRLpowzvqixyUK/v7XoJLnIAfhsfg nuFVtMU9qs+Gk0AZ49qunFN6W5nCAy3GG2qFgHS0grwgBAQDI7Z8tgIL/CKsWQ9V8a9B lUkWrQmoGkWfels9rs4vFz+4uz8LtlVhXZtFA= MIME-Version: 1.0 Received: by 10.216.131.152 with SMTP id m24mr1934372wei.56.1323968198451; Thu, 15 Dec 2011 08:56:38 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 08:56:38 -0800 (PST) In-Reply-To: <4EEA25BB.7040706@sentex.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> <4EEA25BB.7040706@sentex.net> Date: Thu, 15 Dec 2011 17:56:38 +0100 X-Google-Sender-Auth: eN811JKam6VzmdGL06meH8StiGo Message-ID: From: Attilio Rao To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 16:56:40 -0000 2011/12/15 Mike Tancsa : > On 12/15/2011 11:42 AM, Attilio Rao wrote: >> >> I'm thinking now to a better test-case for this: can you try that on a >> tmpfs volume? > > There is enough RAM in the box so that it should not touch the disk, and > I was sending the output to /dev/null, so it was not writing to the disk. > >> >> Also what filesystem you were using? > > UFS > >> How many CPUs were in place? > > 4 > >> Did you reboot before to move the steal_thresh value? > > No. So, as very first thing, can you try the following: - Same codebase, etc. etc. - Make the test 4 times, discard the first and ministat for the other 3 - Reboot - Change the steal_thresh value - Make the test 4 times, discard the first and ministat for the other 3 Then report discarded values and the ministated one and we will have more informations I guess (also, I don't think devfs contention should play a role here, thus nevermind about it for now). Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 17:30:15 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 515A2106566B for ; Thu, 15 Dec 2011 17:30:15 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id F3CFB8FC08 for ; Thu, 15 Dec 2011 17:30:14 +0000 (UTC) Received: by vbbfr13 with SMTP id fr13so2838158vbb.13 for ; Thu, 15 Dec 2011 09:30:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=fxFEcuAaEHP/i6dmW3/ziN3EBmxAV9huu0jQp0UEzc0=; b=qkFuD91IKOooYjt/IM58F0dQrb12W7PfA+pQyjFlxcKB+ntLYxoGjP3ElU1dJLY7G0 +ZLX/HKv2Ozb3fl/aZdn+UX+iIisYZS1Fds/OIPrz6faD3T2K5angs4h6ND3oj0b/Z+b ZP1g7/+F6uJTlKxcuZLvPGMVVQAOXS/9MhKA0= MIME-Version: 1.0 Received: by 10.52.90.80 with SMTP id bu16mr3764448vdb.113.1323970213941; Thu, 15 Dec 2011 09:30:13 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.26.50 with HTTP; Thu, 15 Dec 2011 09:30:13 -0800 (PST) In-Reply-To: <20111215144924.61a08733@elena.home> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <20111215144924.61a08733@elena.home> Date: Thu, 15 Dec 2011 09:30:13 -0800 X-Google-Sender-Auth: RpIQ-wrW5EyFW_lvAQfEWzh6Q_o Message-ID: From: Adrian Chadd To: Tony McC , michael.larabel@phoronix.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 17:30:15 -0000 On 15 December 2011 06:49, Tony McC wrote: > I suggest always ignoring benchmarks. They are like reading the > astrology column in a tabloid newspaper. =A0Instead, try FreeBSD for your > work. =A0Is it fast enough? =A0Surely that is all you need to know. FreeB= SD > is quite fast enough for my needs and I am simply more productive using > it than when I use any other operating system. =A0That is partly to do > with my familiarity with my setup, which I have customised the way I > want. =A0That is something that no benchmark can allow for. You can't ignore benchmarks because: * people read them; * media link to them; * they have pretty pictures. These are all very important things. If all we do is talk on a mailing list and never write public articles of our own, if we never push out our message or work with groups like phronix to dig into the WHY, we're going to be stuck looking bad. It doesn't matter if we aren't bad, we still look bad. It's PR and marketing 101. :) This discussion with Michael @ Phronix is very helpful. Michael, are you willing to help dig into why this is the case, and possibly write a followup article or two about it? I believe Stefan's response looking at what the benchmarks actually do is worthwhile. I wonder if there's a way to get FreeBSD to behave the same at least for comparison, so you can publish what the underlying differences are due to and what relevance they have in the real world. To everyone else (including Jeremy) - don't be afraid to publish and be wrong. You invite discussion and further research that way. Just don't act like a self-absorbed asshat. Everyone benefits. :) Adrian From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 17:58:56 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8061C106566B; Thu, 15 Dec 2011 17:58:56 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id EA6368FC13; Thu, 15 Dec 2011 17:58:55 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RbFa2-0002Al-Qy>; Thu, 15 Dec 2011 18:58:54 +0100 Received: from e178037243.adsl.alicedsl.de ([85.178.37.243] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RbFa2-00087M-Ls>; Thu, 15 Dec 2011 18:58:54 +0100 Message-ID: <4EEA3556.7030105@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 18:58:46 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Daniel Kalchev References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> In-Reply-To: X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig90C2E873E806961B03779663" X-Originating-IP: 85.178.37.243 Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , Stefan Esser , Michael Ross , freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 17:58:56 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig90C2E873E806961B03779663 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Am 12/15/11 14:51, schrieb Daniel Kalchev: >=20 > On Dec 15, 2011, at 3:25 PM, Stefan Esser wrote: >=20 >> Am 15.12.2011 11:10, schrieb Michael Larabel: >>> No, the same hardware was used for each OS. >>> >>> In terms of the software, the stock software stack for each OS was us= ed. >> >> Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with >> journaling enabled) should be an obvious choice since it is more simil= ar >> in concept to ext4 and since that is what most FreeBSD users will use >> with FreeBSD? >=20 >=20 > Or perhaps, since it is "server" Linux distribution, use ZFS on Linux a= s well. With identical tuning on both Linux and FreeBSD. Having the same = FS used by both OS will help make the comparison more sensible for FS I/O= =2E >=20 > Daniel_______________________________________________ Since ZFS in Linux can only be achieved via FUSE (ad far as I know), it is legitimate to compare ZFS and ext4. It would be much more competetive to compare Linux BTRFS and FreeBSD ZFS. Each OS does optimize on different filesystems and a user/manager can assume that the vendor offers the best performance available by turning on the default FS by a standard stock installation. Using ZFS on Linux would be a great disadvantage and the benchmark would turn out the same bullsh... as comparing Linux-domain only with FreeBSD weknesses only ... Linux distributions offer setups for desktop and server. The FreeBSD folks have the choice to do it themselfes. And maybe I'm one of those puritain people appreciating this. "Out of the box" OS is Windooze, with all its consequences. Oliver Post scriptum: It seems to be hard to follow the benchmark environment on Phoronix since the URL refers to a setup of different systems. --------------enig90C2E873E806961B03779663 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6jVeAAoJEOgBcD7A/5N8ALYH/0un2B7HHTHdeoxEzN9UJ8x+ WhlqiupymlpJR2UJDkWlDRETa9JABFE6Iuc84iAPbcHExzyd6BbYMhr9pvX0OlCM p1IWXUHrpXzr3fs3qoWtQIJi4yr6/Wb2dvJJHBK8tuwyOd2XR9GC4/sIwFXpw7Up 7343FJKIhZXpacEkJP/rQz9PxjcmifzuuDQhwjvrbTJDoJn5CQvya9gcVFExlTBS 2qHd7UwCjRf6xiu9lhTEtYy4O5uZoqSLeprxTEowP4DRwSbBJ33Ix0eAGmEc4vfB OnijWNZVnR4J8VrSj1DXltb8t/wTKe9bWzT8lVf98cVK018vQ2h3juiASY/y++A= =6R/p -----END PGP SIGNATURE----- --------------enig90C2E873E806961B03779663-- From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 18:00:47 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5810D106566C; Thu, 15 Dec 2011 18:00:46 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 02E8C8FC1A; Thu, 15 Dec 2011 18:00:45 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RbFbo-0002Sq-J3>; Thu, 15 Dec 2011 19:00:44 +0100 Received: from e178037243.adsl.alicedsl.de ([85.178.37.243] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RbFbo-0008Dz-DT>; Thu, 15 Dec 2011 19:00:44 +0100 Message-ID: <4EEA35CB.7030203@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 19:00:43 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Daniel Kalchev References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <20111215134853.GA24753@icarus.home.lan> <0C72D682-CF5E-42D6-91F3-FEF1AB02F5D6@digsys.bg> In-Reply-To: <0C72D682-CF5E-42D6-91F3-FEF1AB02F5D6@digsys.bg> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigB97FEA6B9A0C0983989EA20A" X-Originating-IP: 85.178.37.243 Cc: Adrian Chadd , FreeBSD Stable Mailing List , Current FreeBSD , "Samuel J. Greear" , freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 18:00:47 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB97FEA6B9A0C0983989EA20A Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Am 12/15/11 14:58, schrieb Daniel Kalchev: >=20 > On Dec 15, 2011, at 3:48 PM, Jeremy Chadwick wrote: >=20 > [=85] >> That said: thrown out, data ignored, done. >> >> Now what? Where are we? We're right back where we were a day or two >> ago; meaning no closer to solving the dilemma reported by users and >> SCHED_ULE. Heck, we're not even sure if there is an issue, other than= >> some folks confirming that SCHED_4BSD performs better for them (that's= >> what started this whole thread), and there are at least a couple which= >> have stated this. >=20 > But, are any of these benchmarks really engaging the 4BSD/ULE scheduler= differences? Most such benchmarks are run on a system with no other load= whatsoever and in no way represent real world experience. >=20 > What is more, I believe in such benchmarks "the system feels sluggish" = is not measured at all. Even if it is measured, if in such case the bench= mark finishes "better" - that is, faster, or say, makes the system freeze= for the user for the duration of the test -- it will be considered "win"= , because the benchmark suite ran faster on that particular system -- whe= reas a system which ran the benchmark fast, provided good interactive res= ponse etc would be considered "loser". I guess you have some proofs on that "feeling"? >=20 > I think it is not good idea to hijack this thread, but instead focusing= on the other SCHED_ULE bashing thread to define an reasonable benchmark = or a set of benchmarks rather -- so that many would run it and provide fe= edback. >=20 >=20 > Daniel_______________________________________________ --------------enigB97FEA6B9A0C0983989EA20A Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6jXLAAoJEOgBcD7A/5N8eHMH+gIP8qH2klzQvMwrpp40QhU1 E1Bd4Q13P5RAc69oJJWdBzz4jV9Oz9aJZzpc4uHnFI9FyxBVY9LL3QVuX3cErK7u NmxS6Hl3AkrfAZ2I0O/XGq6LF6Kmcw83LCKWubexRAaIIr4YjZd/AiTd5TlU1nyy Nml9b8yyJlt9aggS22TO6UTnqRxcvqFQhP8hAZnPjYsoN6sDd3TRynAJqNc7LWeW P8jBxo2+gqEnNDl4LYrr+RDM6Gsbr3k2+YYK98miX/DUHBLEBx0liVCpy+lPNWhl XBqGGGPjveUqBEVvUiOixU7aO8rxQDnL3PBSdreL7xeOTvP9bRdZ1lnaxpEq+4M= =YwYJ -----END PGP SIGNATURE----- --------------enigB97FEA6B9A0C0983989EA20A-- From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 18:06:23 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 27C6E1065690; Thu, 15 Dec 2011 18:06:23 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9C5F18FC17; Thu, 15 Dec 2011 18:06:22 +0000 (UTC) Received: by vcbfk1 with SMTP id fk1so3005421vcb.13 for ; Thu, 15 Dec 2011 10:06:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2AlMjn6CxxbBgoVYL7CStHaRl2qG88b4zva2K1TTroc=; b=IozSrMog/mt1j+wgGQuRVp4rgnds8/VYI9At0SDz4wIHpfLqULQUoPLI1ilruuxVce 0r3t81IdSDmasQoqkCRadSxHGWq5SxFR3NHvxsQQXGnqS4wQ5fXJYkcE21kFzvjszSA+ 4Gfle2IoCwqR2TWDlNLapfsqZ9442wwCeb4ww= MIME-Version: 1.0 Received: by 10.220.148.146 with SMTP id p18mr880503vcv.46.1323972381673; Thu, 15 Dec 2011 10:06:21 -0800 (PST) Received: by 10.220.231.10 with HTTP; Thu, 15 Dec 2011 10:06:21 -0800 (PST) In-Reply-To: <4EEA3556.7030105@zedat.fu-berlin.de> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EE9C79B.7080607@phoronix.com> <4EE9F546.6060503@freebsd.org> <4EEA3556.7030105@zedat.fu-berlin.de> Date: Thu, 15 Dec 2011 10:06:21 -0800 Message-ID: From: Freddie Cash To: "O. Hartmann" Content-Type: text/plain; charset=UTF-8 Cc: Michael Larabel , FreeBSD Stable Mailing List , Current FreeBSD , Daniel Kalchev , Michael Ross , freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 18:06:23 -0000 On Thu, Dec 15, 2011 at 9:58 AM, O. Hartmann wrote: > Am 12/15/11 14:51, schrieb Daniel Kalchev: >> >> On Dec 15, 2011, at 3:25 PM, Stefan Esser wrote: >> >>> Am 15.12.2011 11:10, schrieb Michael Larabel: >>>> No, the same hardware was used for each OS. >>>> >>>> In terms of the software, the stock software stack for each OS was used. >>> >>> Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with >>> journaling enabled) should be an obvious choice since it is more similar >>> in concept to ext4 and since that is what most FreeBSD users will use >>> with FreeBSD? >> >> >> Or perhaps, since it is "server" Linux distribution, use ZFS on Linux as well. With identical tuning on both Linux and FreeBSD. Having the same FS used by both OS will help make the comparison more sensible for FS I/O. >> >> Daniel_______________________________________________ > > Since ZFS in Linux can only be achieved via FUSE (ad far as I know), it > is legitimate to compare ZFS and ext4. It would be much more competetive > to compare Linux BTRFS and FreeBSD ZFS. There is a separate kernel module for ZFS that can be installed, giving you proper kernel-level support for ZFS on Linux. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 19:02:48 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21D4E106564A; Thu, 15 Dec 2011 19:02:48 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4B6198FC13; Thu, 15 Dec 2011 19:02:45 +0000 (UTC) Received: by faaf16 with SMTP id f16so3819416faa.13 for ; Thu, 15 Dec 2011 11:02:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=SgZAVHUFCa9dzNQ7v791Vzanwzpvg9L2LKkf09pCqIA=; b=dKua/AxeaedYAKKhiqDkk6XJxGID2B/hN+gJs34qdsLEwXFhEbp8i6LudZbQ24sAiC emCaA+2CgFaBfDoGho7LhILfS1yOSBqRKyrFY441CQIwjKkM2OZlTWueIClf4uVwua1j AO1bSgwuR+b01OV9+sKbXUZ7dMjZuPkDAQzxE= MIME-Version: 1.0 Received: by 10.180.96.103 with SMTP id dr7mr7874751wib.16.1323975764332; Thu, 15 Dec 2011 11:02:44 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 11:02:44 -0800 (PST) In-Reply-To: <20111215174857.GA28551@icarus.home.lan> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> <20111215174857.GA28551@icarus.home.lan> Date: Thu, 15 Dec 2011 20:02:44 +0100 X-Google-Sender-Auth: CHfHol8uWmtjFqXYPz0qUtT3ARA Message-ID: From: Attilio Rao To: Jeremy Chadwick Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 19:02:48 -0000 2011/12/15 Jeremy Chadwick : > On Thu, Dec 15, 2011 at 05:26:27PM +0100, Attilio Rao wrote: >> 2011/12/13 Jeremy Chadwick : >> > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: >> >> > Not fully right, boinc defaults to run on idprio 31 so this isn't a= n >> >> > issue. And yes, there are cases where SCHED_ULE shows much better >> >> > performance then SCHED_4BSD. ??[...] >> >> >> >> Do we have any proof at hand for such cases where SCHED_ULE performs >> >> much better than SCHED_4BSD? Whenever the subject comes up, it is >> >> mentioned, that SCHED_ULE has better performance on boxes with a ncpu= > >> >> 2. But in the end I see here contradictionary statements. People >> >> complain about poor performance (especially in scientific environment= s), >> >> and other give contra not being the case. >> >> >> >> Within our department, we developed a highly scalable code for planet= ary >> >> science purposes on imagery. It utilizes present GPUs via OpenCL if >> >> present. Otherwise it grabs as many cores as it can. >> >> By the end of this year I'll get a new desktop box based on Intels ne= w >> >> Sandy Bridge-E architecture with plenty of memory. If the colleague w= ho >> >> developed the code is willing performing some benchmarks on the same >> >> hardware platform, we'll benchmark bot FreeBSD 9.0/10.0 and the most >> >> recent Suse. For FreeBSD I intent also to look for performance with b= oth >> >> different schedulers available. >> > >> > This is in no way shape or form the same kind of benchmark as what >> > you're planning to do, but I thought I'd throw it out there for folks = to >> > take in as they see fit. >> > >> > I know folks were focused mainly on buildworld. >> > >> > I personally would find it interesting if someone with a higher-end >> > system (e.g. 2 physical CPUs, with 6 or 8 cores per CPU) was to do the >> > same test (changing -jX to -j{numofcores} of course). >> > >> > -- >> > | Jeremy Chadwick ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??jdc a= t parodius.com | >> > | Parodius Networking ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? http://www.paro= dius.com/ | >> > | UNIX Systems Administrator ?? ?? ?? ?? ?? ?? ?? ?? ?? Mountain View,= CA, US | >> > | Making life hard for others since 1977. ?? ?? ?? ?? ?? ?? ?? PGP 4BD= 6C0CB | >> > >> > >> > sched_ule >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> > - time make -j2 buildworld >> > ??1689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io 4565pf+0w >> > - time make -j2 buildkernel >> > ??640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w >> > >> > >> > sched_4bsd >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> > - time make -j2 buildworld >> > ??1662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io 6451pf+0w >> > - time make -j2 buildkernel >> > ??638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w >> > >> > >> > software >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> > * sched_ule test: ??FreeBSD 8.2-STABLE, Thu Dec ??1 04:37:29 PST 2011 >> > * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST 2011 >> >> Hi Jeremy, >> thanks for the time you spent on this. >> >> However, I wanted to ask/let you note 3 things: >> 1) Did you use 2 different code base for the test? (one updated on >> December 1 and another one on December 12) > > No; src-all (/usr/src on this system) was not updated between December > 1st and December 12th PST. =C2=A0I do believe I updated it today (15th PS= T). > I can/will obviously hold off so that we have a consistent code base for > comparing numbers between schedulers during buildworld and/or > buildkernel. > >> 2) Please note that you should have repeated this test several times >> (basically until you don't get a standard deviation which is >> acceptable with ministat) and report the ministat output > > This is the first time I have heard of ministat(1). =C2=A0I'm pretty sure= I > see what it's for and how it applies to this situation, but boy that man > page could use some clarification (I have 3 people looking at this thing > right now trying to figure out what means what in the graph :-) ). > Anyway, graph or not, I see the point. > > Regarding multiple tests: yup, you're absolutely right, the only way to > do it would be to run a sequence of tests repeatedly (probably 10 per > scheduler). =C2=A0Reboots and rm -fr /usr/obj/* would be required after e= ach > test too, to guarantee empty kernel caches (of all types) consistently > every time. > > What I posted was supposed to give people just a "general idea" if there > was any gigantic difference between the two, and there really isn't. > But, as others have stated (and you below), buildworld may not be an > effective way to "benchmark" what we're trying to test. > > Hence me wondering exactly what would make for a good test. =C2=A0Example= : > > 1. Run + background some program that "beats on things" (I really don't > know what; creation/deletion of threads? =C2=A0CPU benchmark? =C2=A0bonni= e++?), > with output going to /dev/null. > 2. Run + background "time make -j2 buildworld" with output going to /dev/= null > 3. Record/save output from "time". > 4. rm -fr /usr/obj && shutdown -r now > 5. Repeat all steps ~10 times > 6. Adjust kernel configuration file to use other scheduler > 7. Repeat steps 1-5. > > What I'm trying to figure out is what #1 and #2 should be in the above > example. > >> 3) The difference is less than 2% which I suspect is really >> statistically unuseful/the same > > Understood. > >> I'm not really even surprised ULE is not faster than 4BSD in this case >> because usually buildworld/buildkernel tests are driven for the vast >> majority by I/O overhead rather than scheduler capacity. It would be >> more interesting to analyze how buildworld does while another type of >> workload is going on. > > Yup, agreed/understood, hence me trying to find out what would classify > as a good stress test for all of this. > > I have a testbed system in my garage which I could set up to literally > do all of this in a loop, meaning automate the entire above process and > just let it go, writing stderr from time to a file (which wouldn't skew > the results at all). > > Let me know what #1 and #2 above, re: "the workloads", should be and > I'll be happy to set it up. My idea, in order to gather meaningful datas for both ULE and 4BSD would be to see how well they behave in the futher situation: - 2 concurrent interactive workloads - 2 concurrent cpu-intensive workloads - mixed and having the number of threads for both varying as: N/2, N, N + small_amount (1 or 2 or 3, etc), N*2 (where N is the number of available CPUs) which automatically translates into: - 2 concurrent interactive and intensive (A and B workloads): * A N/2 threads, B N/2 threads * A N threads, B N/2 threads * A N + small_amount, B N/2 threads * A N*2 threads, B N/2 threads * A N threads, B N threads * A N + small_amount, B N threads * A N*2 threads, B N threads * A N + small_amount, B N + small_amount threads * A N*2 threads, B N + small_amount threads * A N*2 threads, B N*2 threads For the mixed case, instead, we should try all the 16 combinations possibly and it is likely the most interesting case, to be honest. About the workload, we could use: interactives: buildworld and bonnie++ (I'm not totally sure if bonnie++ let you decides how many threads to run, but I'm sure we can replace with something that really does that) cpu-intensive: dnetc and SOMETHINGELSE (please propose something that can be setup very easilly!) mixed case: buildworld and dnetc About the environment I'd suggest the following things: - Try to boot with a maximum of 16 CPUs. I'm sure past that point TLB shootdown overhead is going to be too overwhelming, make doesn't really scale well, and also there could be too much contention on vm_page_lock_queue for interactive threads. - Try to reduce the I/O effect by using tmpfs as a storage for in and out datas when working out the benchmark - Use 10.0 with both kerneland and userland totally debug-free (please recall to set MALLOC_PRODUCTION in jemalloc) and always at the same svn revision, with the only change being the scheduler switch and the number of threads changing during the runs About the test itself I'd suggest the following things: - After every test combination, please reboot the machine (like, after you have tested the A N/2 threads and B N/2 threads case on sched_4bsd, reboot the machine before to do A N threads and B N/2 threads) - For every test combination I suggest to run the workloads 4 times, discard the first one (but keep the value!) and ministat the other three. Showing the "uncached" case against the average cached one will give much more indication than expected. - Expect a standard deviation from ministat to be 95% (or beyond) to be val= uable - For every difference in performance we find we should likely start worry about if it is as or bigger than 3% and being very concerned from 5% to above I think we already have some datas of ULE being broken in some cases (like George's and Steven's case) but we really need to characterize more, I think. Now, I understand this seems a gigantic work but I think there is much people which is interested in working on this and we may scatter these tests around, to different testers, to find meaningful datas. If it was me, I would start with comparisons involving all the N and N + small_amount cases which should be the most interesting. Do you have questions? Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 19:46:36 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB89C106564A; Thu, 15 Dec 2011 19:46:36 +0000 (UTC) (envelope-from fidaj@ukr.net) Received: from fsm2.ukr.net (fsm2.ukr.net [195.214.192.121]) by mx1.freebsd.org (Postfix) with ESMTP id 6820F8FC0A; Thu, 15 Dec 2011 19:46:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=fsm; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-ID:Subject:Cc:To:From:Date; bh=EK+mEIr5wXLNRK+o+gkQwhNeZz7gH4y//yqgkqZMT9E=; b=ClJh5RpI1ShemCy0/yCT1WdqtbQbx66RQR01kStLiG7jjpjg0o65EgTq+l9kHG5f9+Lqoj3t6KYv0fWIB2MseglX6MT+mqfCfOH76g3f/DV83LKXtbLlAFtKDILOT7cLpZNA/x4v5v3vlcozZxllSG5ElG/AKj7dHzTvN4DJ7SM=; Received: from [178.137.138.140] (helo=nonamehost.) by fsm2.ukr.net with esmtpsa ID 1RbHGD-000ME3-L6 ; Thu, 15 Dec 2011 21:46:33 +0200 Date: Thu, 15 Dec 2011 21:46:27 +0200 From: Ivan Klymenko To: Attilio Rao Message-ID: <20111215214627.16f472bf@nonamehost.> In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <20111213073615.GA69641@icarus.home.lan> <20111215174857.GA28551@icarus.home.lan> X-Mailer: Claws Mail 3.7.10 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "O. Hartmann" , Current FreeBSD , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, Jeremy Chadwick Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 19:46:37 -0000 =D0=92 Thu, 15 Dec 2011 20:02:44 +0100 Attilio Rao =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > 2011/12/15 Jeremy Chadwick : > > On Thu, Dec 15, 2011 at 05:26:27PM +0100, Attilio Rao wrote: > >> 2011/12/13 Jeremy Chadwick : > >> > On Mon, Dec 12, 2011 at 02:47:57PM +0100, O. Hartmann wrote: > >> >> > Not fully right, boinc defaults to run on idprio 31 so this > >> >> > isn't an issue. And yes, there are cases where SCHED_ULE > >> >> > shows much better performance then SCHED_4BSD. ??[...] > >> >> > >> >> Do we have any proof at hand for such cases where SCHED_ULE > >> >> performs much better than SCHED_4BSD? Whenever the subject > >> >> comes up, it is mentioned, that SCHED_ULE has better > >> >> performance on boxes with a ncpu > 2. But in the end I see here > >> >> contradictionary statements. People complain about poor > >> >> performance (especially in scientific environments), and other > >> >> give contra not being the case. > >> >> > >> >> Within our department, we developed a highly scalable code for > >> >> planetary science purposes on imagery. It utilizes present GPUs > >> >> via OpenCL if present. Otherwise it grabs as many cores as it > >> >> can. By the end of this year I'll get a new desktop box based > >> >> on Intels new Sandy Bridge-E architecture with plenty of > >> >> memory. If the colleague who developed the code is willing > >> >> performing some benchmarks on the same hardware platform, we'll > >> >> benchmark bot FreeBSD 9.0/10.0 and the most recent Suse. For > >> >> FreeBSD I intent also to look for performance with both > >> >> different schedulers available. > >> > > >> > This is in no way shape or form the same kind of benchmark as > >> > what you're planning to do, but I thought I'd throw it out there > >> > for folks to take in as they see fit. > >> > > >> > I know folks were focused mainly on buildworld. > >> > > >> > I personally would find it interesting if someone with a > >> > higher-end system (e.g. 2 physical CPUs, with 6 or 8 cores per > >> > CPU) was to do the same test (changing -jX to -j{numofcores} of > >> > course). > >> > > >> > -- > >> > | Jeremy > >> > Chadwick ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??jdc at > >> > parodius.com | | Parodius > >> > Networking ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? > >> > http://www.parodius.com/ | | UNIX Systems > >> > Administrator ?? ?? ?? ?? ?? ?? ?? ?? ?? Mountain View, CA, US | > >> > | Making life hard for others since 1977. ?? ?? ?? ?? ?? ?? ?? > >> > PGP 4BD6C0CB | > >> > > >> > > >> > sched_ule > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> > - time make -j2 buildworld > >> > ??1689.831u 229.328s 18:46.20 170.4% 6566+2051k 432+4264io > >> > 4565pf+0w > >> > - time make -j2 buildkernel > >> > ??640.542u 87.737s 9:01.38 134.5% 6490+1920k 134+5968io 0pf+0w > >> > > >> > > >> > sched_4bsd > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> > - time make -j2 buildworld > >> > ??1662.793u 206.908s 17:12.02 181.1% 6578+2054k 23750+4271io > >> > 6451pf+0w > >> > - time make -j2 buildkernel > >> > ??638.717u 76.146s 8:34.90 138.8% 6530+1927k 6415+5903io 0pf+0w > >> > > >> > > >> > software > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> > * sched_ule test: ??FreeBSD 8.2-STABLE, Thu Dec ??1 04:37:29 PST > >> > 2011 > >> > * sched_4bsd test: FreeBSD 8.2-STABLE, Mon Dec 12 22:42:54 PST > >> > 2011 > >> > >> Hi Jeremy, > >> thanks for the time you spent on this. > >> > >> However, I wanted to ask/let you note 3 things: > >> 1) Did you use 2 different code base for the test? (one updated on > >> December 1 and another one on December 12) > > > > No; src-all (/usr/src on this system) was not updated between > > December 1st and December 12th PST. =C2=A0I do believe I updated it > > today (15th PST). I can/will obviously hold off so that we have a > > consistent code base for comparing numbers between schedulers > > during buildworld and/or buildkernel. > > > >> 2) Please note that you should have repeated this test several > >> times (basically until you don't get a standard deviation which is > >> acceptable with ministat) and report the ministat output > > > > This is the first time I have heard of ministat(1). =C2=A0I'm pretty > > sure I see what it's for and how it applies to this situation, but > > boy that man page could use some clarification (I have 3 people > > looking at this thing right now trying to figure out what means > > what in the graph :-) ). Anyway, graph or not, I see the point. > > > > Regarding multiple tests: yup, you're absolutely right, the only > > way to do it would be to run a sequence of tests repeatedly > > (probably 10 per scheduler). =C2=A0Reboots and rm -fr /usr/obj/* would > > be required after each test too, to guarantee empty kernel caches > > (of all types) consistently every time. > > > > What I posted was supposed to give people just a "general idea" if > > there was any gigantic difference between the two, and there really > > isn't. But, as others have stated (and you below), buildworld may > > not be an effective way to "benchmark" what we're trying to test. > > > > Hence me wondering exactly what would make for a good test. > > =C2=A0Example: > > > > 1. Run + background some program that "beats on things" (I really > > don't know what; creation/deletion of threads? =C2=A0CPU benchmark? > > =C2=A0bonnie++?), with output going to /dev/null. > > 2. Run + background "time make -j2 buildworld" with output going > > to /dev/null 3. Record/save output from "time". > > 4. rm -fr /usr/obj && shutdown -r now > > 5. Repeat all steps ~10 times > > 6. Adjust kernel configuration file to use other scheduler > > 7. Repeat steps 1-5. > > > > What I'm trying to figure out is what #1 and #2 should be in the > > above example. > > > >> 3) The difference is less than 2% which I suspect is really > >> statistically unuseful/the same > > > > Understood. > > > >> I'm not really even surprised ULE is not faster than 4BSD in this > >> case because usually buildworld/buildkernel tests are driven for > >> the vast majority by I/O overhead rather than scheduler capacity. > >> It would be more interesting to analyze how buildworld does while > >> another type of workload is going on. > > > > Yup, agreed/understood, hence me trying to find out what would > > classify as a good stress test for all of this. > > > > I have a testbed system in my garage which I could set up to > > literally do all of this in a loop, meaning automate the entire > > above process and just let it go, writing stderr from time to a > > file (which wouldn't skew the results at all). > > > > Let me know what #1 and #2 above, re: "the workloads", should be and > > I'll be happy to set it up. >=20 > My idea, in order to gather meaningful datas for both ULE and 4BSD > would be to see how well they behave in the futher situation: > - 2 concurrent interactive workloads > - 2 concurrent cpu-intensive workloads > - mixed >=20 > and having the number of threads for both varying as: N/2, N, N + > small_amount (1 or 2 or 3, etc), N*2 (where N is the number of > available CPUs) which automatically translates into: >=20 > - 2 concurrent interactive and intensive (A and B workloads): > * A N/2 threads, B N/2 threads > * A N threads, B N/2 threads > * A N + small_amount, B N/2 threads > * A N*2 threads, B N/2 threads > * A N threads, B N threads > * A N + small_amount, B N threads > * A N*2 threads, B N threads > * A N + small_amount, B N + small_amount threads > * A N*2 threads, B N + small_amount threads > * A N*2 threads, B N*2 threads >=20 > For the mixed case, instead, we should try all the 16 combinations > possibly and it is likely the most interesting case, to be honest. >=20 > About the workload, we could use: > interactives: buildworld and bonnie++ (I'm not totally sure if > bonnie++ let you decides how many threads to run, but I'm sure we can > replace with something that really does that) > cpu-intensive: dnetc and SOMETHINGELSE (please propose something that > can be setup very easilly!) > mixed case: buildworld and dnetc >=20 > About the environment I'd suggest the following things: > - Try to boot with a maximum of 16 CPUs. I'm sure past that point TLB > shootdown overhead is going to be too overwhelming, make doesn't > really scale well, and also there could be too much contention on > vm_page_lock_queue for interactive threads. > - Try to reduce the I/O effect by using tmpfs as a storage for in and > out datas when working out the benchmark > - Use 10.0 with both kerneland and userland totally debug-free (please > recall to set MALLOC_PRODUCTION in jemalloc) and always at the same > svn revision, with the only change being the scheduler switch and the > number of threads changing during the runs >=20 > About the test itself I'd suggest the following things: > - After every test combination, please reboot the machine (like, after > you have tested the A N/2 threads and B N/2 threads case on > sched_4bsd, reboot the machine before to do A N threads and B N/2 > threads) > - For every test combination I suggest to run the workloads 4 times, > discard the first one (but keep the value!) and ministat the other > three. Showing the "uncached" case against the average cached one will > give much more indication than expected. > - Expect a standard deviation from ministat to be 95% (or beyond) to > be valuable > - For every difference in performance we find we should likely start > worry about if it is as or bigger than 3% and being very concerned > from 5% to above >=20 > I think we already have some datas of ULE being broken in some cases > (like George's and Steven's case) but we really need to characterize > more, I think. >=20 > Now, I understand this seems a gigantic work but I think there is much > people which is interested in working on this and we may scatter these > tests around, to different testers, to find meaningful datas. >=20 > If it was me, I would start with comparisons involving all the N and N > + small_amount cases which should be the most interesting. >=20 > Do you have questions? >=20 > Thanks, > Attilio >=20 >=20 Perhaps it makes sense to co-write a script to automate these actions? And place it in /usr/src/tools/sched/... From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 20:58:07 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBB52106564A; Thu, 15 Dec 2011 20:58:07 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1-6.sentex.ca [IPv6:2607:f3e0:0:1::12]) by mx1.freebsd.org (Postfix) with ESMTP id 90BBC8FC12; Thu, 15 Dec 2011 20:58:07 +0000 (UTC) Received: from [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a] (saphire3.sentex.ca [IPv6:2607:f3e0:0:4:f025:8813:7603:7e4a]) by smarthost1.sentex.ca (8.14.5/8.14.4) with ESMTP id pBFKw2ea013553; Thu, 15 Dec 2011 15:58:03 -0500 (EST) (envelope-from mike@sentex.net) Message-ID: <4EEA5F5C.8080503@sentex.net> Date: Thu, 15 Dec 2011 15:58:04 -0500 From: Mike Tancsa Organization: Sentex Communications User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Attilio Rao References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> <4EEA25BB.7040706@sentex.net> In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.71 on IPv6:2607:f3e0:0:1::12 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 20:58:08 -0000 On 12/15/2011 11:56 AM, Attilio Rao wrote: > So, as very first thing, can you try the following: > - Same codebase, etc. etc. > - Make the test 4 times, discard the first and ministat for the other 3 > - Reboot > - Change the steal_thresh value > - Make the test 4 times, discard the first and ministat for the other 3 > > Then report discarded values and the ministated one and we will have > more informations I guess > (also, I don't think devfs contention should play a role here, thus > nevermind about it for now). Results and data at http://www.tancsa.com/ule-bsd.html ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ From owner-freebsd-performance@FreeBSD.ORG Thu Dec 15 21:30:11 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C4421065675; Thu, 15 Dec 2011 21:30:11 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4C5D98FC12; Thu, 15 Dec 2011 21:30:09 +0000 (UTC) Received: by werb13 with SMTP id b13so308398wer.13 for ; Thu, 15 Dec 2011 13:30:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=n7KfzfVyFNQXWNxFW6vjVaEbwJSYP2hpwEXCEg172hY=; b=JeYX+K+bT1XoVpq9v5x0AF2oW1Z7WZgDunB3J3Ss1LT9ZreedL1C7sd//L+D/rb/Ny WaQXzTbYVuAwiXTS6rxgJlwlkfB4EtRzyjEJnLIYxLiD0npcroxDSutPkDzVDVCuPz5X VVriBCK/hiC2kvAcyATssUPls8PpRMJMCenyI= MIME-Version: 1.0 Received: by 10.216.131.152 with SMTP id m24mr2521388wei.56.1323984609072; Thu, 15 Dec 2011 13:30:09 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Thu, 15 Dec 2011 13:30:08 -0800 (PST) In-Reply-To: <4EEA5F5C.8080503@sentex.net> References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost.> <20111213230441.GB42285@stack.nl> <4ee7e2d3.0a3c640a.4617.4a33SMTPIN_ADDED@mx.google.com> <4EE8D607.1000504@sentex.net> <4EEA227E.7080704@sentex.net> <4EEA25BB.7040706@sentex.net> <4EEA5F5C.8080503@sentex.net> Date: Thu, 15 Dec 2011 22:30:08 +0100 X-Google-Sender-Auth: -u6X1A6ozE_YndJdHRmklZbhh6k Message-ID: From: Attilio Rao To: Mike Tancsa Content-Type: text/plain; charset=UTF-8 Cc: Ivan Klymenko , mdf@freebsd.org, Doug Barton , freebsd-stable@freebsd.org, Jilles Tjoelker , "O. Hartmann" , Current FreeBSD , freebsd-performance@freebsd.org Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 21:30:11 -0000 2011/12/15 Mike Tancsa : > On 12/15/2011 11:56 AM, Attilio Rao wrote: >> So, as very first thing, can you try the following: >> - Same codebase, etc. etc. >> - Make the test 4 times, discard the first and ministat for the other 3 >> - Reboot >> - Change the steal_thresh value >> - Make the test 4 times, discard the first and ministat for the other 3 >> >> Then report discarded values and the ministated one and we will have >> more informations I guess >> (also, I don't think devfs contention should play a role here, thus >> nevermind about it for now). > > > Results and data at > > http://www.tancsa.com/ule-bsd.html I'm not totally sure, what does burnP6 do? is it a CPU-bound workload? Also, how many threads are spanked in your case for parallel bzip2? Also, it would be very good if you could arrange these tests against newer -CURRENT (with userland and kerneland debugging off). Thanks a lot of your hard work, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-performance@FreeBSD.ORG Fri Dec 16 07:06:13 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01B791065672; Fri, 16 Dec 2011 07:06:13 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id A11568FC08; Fri, 16 Dec 2011 07:06:12 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RbRru-0006Wn-Rh>; Fri, 16 Dec 2011 08:06:10 +0100 Received: from e178018085.adsl.alicedsl.de ([85.178.18.85] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RbRru-00022J-Md>; Fri, 16 Dec 2011 08:06:10 +0100 Message-ID: <4EEAEDE1.50604@zedat.fu-berlin.de> Date: Fri, 16 Dec 2011 08:06:09 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Joe Holden References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EEAE8DF.40303@rewt.org.uk> In-Reply-To: <4EEAE8DF.40303@rewt.org.uk> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig39A77F392D735EB75795DD67" X-Originating-IP: 85.178.18.85 Cc: freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List , Jeremy Chadwick , Arnaud Lacombe Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 07:06:13 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig39A77F392D735EB75795DD67 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/16/11 07:44, Joe Holden wrote: > Arnaud Lacombe wrote: >> Hi, >> >> On Thu, Dec 15, 2011 at 2:32 AM, O. Hartmann >> wrote: >>> Just saw this shot benchmark on Phoronix dot com today: >>> >>> http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTAyNzA >>> >> it might be worth highlighting that despite Oracle Linux 6.1 Server is= >> using a kernel + compiler almost 2 years old, it still manages to >> out-perform the bleeding edge FreeBSD :-) >> > serenity# gcc --version > gcc (GCC) 4.2.1 20070831 patched [FreeBSD] >=20 > serenity# uname -r > 9.0-RC3 >=20 For the underlying OS, as far as I know, the compiler hasn't as much impact as on userland software since autovectorization and other neat things are not used during system build. =46rom my experience using gcc 4.2 or 4.4/4.5 does not have an impact beyond 3% when SSE isn't explicetly enforced. More interesting is the performance gain due to the architecture. I think it would be very easy for M. Larabel to repeat this benchmark with a "bleeding edge" Ubuntu or Suse as well. And since FreeBSD 9.0 can be compiled with CLANG, it should be possible to compare both also with "bleeding edge" compilers, say FreeBSD 9/CLANG, Ubuntu 12/gcc 4.6.2. >> Now, from what I've read so far in this thread, it seems that a lot of= >> people are still in abnegation... >> >> my 0.2c, >> - Arnaud >> >>> It may be worth to discuss the sad performance of FBSD in some parts = of >>> the benchmark. A difference of a factor 10 or 100 is simply far beyon= d >>> disapointing, it is more than inacceptable and by just reading those >>> benchmarks, I'd like to drop thinking of using FreeBSD even as a back= end >>> server in scientific and business environments. In detail, some of th= e >>> SciMark benches look disappointing. The overall image can't help over= >>> the fact that in C-Ray FreeBSD is better performing. >>> >>> From the compiler, I'd like say there couldn't be a drop of more than= 10 >>> - 15% in performance - but not 10 or 100 times. >>> >>> I'm just thinking about the discussion of SCHED_ULE and all the saur >>> spots we discussed when I stumbled over the test. >>> >>> Regards, >>> Oliver --------------enig39A77F392D735EB75795DD67 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO6u3hAAoJEOgBcD7A/5N8TCsH/1Aqj3f2K3XC7u6sDI8GlXn6 OK0v5A6UrlGHHWGz6mrJ60EeH8406T/e2eA1E0iRJotQwGdr0Rvpcm+J0bxcqom8 uBVJ/yXLFyiGT3GZR7t27/wrTRXRV9yYlxqaYs9zvTf2e9rUO4ttqx69yNV5SuSI 9wzkqqA8AmctorRrpyj2wVt0iNUFzFFPSBz/REj9vJOjdFPGdqWJwKUVDeEBQrny q/4lZjhmNX5qeeC2/enceYRgN3FjeYjSqHJ2JHw7qKPnYWYF3r7/J7A0A/es2G6b KQjBTs5xfUvnVwyu2gCQoFpNQ92S5kiq6KDZs+RQ6jaQUxBrEwdWHruOGLj2OIg= =+AbL -----END PGP SIGNATURE----- --------------enig39A77F392D735EB75795DD67-- From owner-freebsd-performance@FreeBSD.ORG Fri Dec 16 10:55:00 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EA53106566B; Fri, 16 Dec 2011 10:55:00 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 62C2A8FC1A; Fri, 16 Dec 2011 10:54:59 +0000 (UTC) Received: by wgbds13 with SMTP id ds13so2866781wgb.1 for ; Fri, 16 Dec 2011 02:54:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=XyEIxMMq4/9l31BQXKMWLuyKLUZiqoXczwT5f7NDMno=; b=s67ButLlyjpBdJwR5ttaf2OHIVq7EQWEGzQavWNOJ4GZHm72+SeTTVSnnkmhC9ejV5 E0DNl11XJyHwxn9Sn7ElEAGLAcVZtDLxYp+IDSG8OIJpfUoTNbZNtspTwsdR8g9V6k17 9oKlUxCrt/xqvDWZFv4Fy6iIZENzGGgLYTbyw= MIME-Version: 1.0 Received: by 10.180.74.211 with SMTP id w19mr11794028wiv.7.1324032896461; Fri, 16 Dec 2011 02:54:56 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.171.8 with HTTP; Fri, 16 Dec 2011 02:54:55 -0800 (PST) In-Reply-To: References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> Date: Fri, 16 Dec 2011 11:54:55 +0100 X-Google-Sender-Auth: KA3O_OU4bWP9K1T7FdTNQyuu6uY Message-ID: From: Attilio Rao To: Arnaud Lacombe Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: FreeBSD Stable Mailing List , freebsd-performance@freebsd.org, Current FreeBSD , "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 10:55:00 -0000 2011/12/16 Arnaud Lacombe : > Hi, > > On Thu, Dec 15, 2011 at 2:32 AM, O. Hartmann > wrote: >> Just saw this shot benchmark on Phoronix dot com today: >> >> http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTAyNzA >> > it might be worth highlighting that despite Oracle Linux 6.1 Server is > using a kernel + compiler almost 2 years old, it still manages to > out-perform the bleeding edge FreeBSD :-) > > Now, from what I've read so far in this thread, it seems that a lot of > people are still in abnegation... > > my 0.2c, > =C2=A0- Arnaud Said by someone which really thinks passing __FILE__ and __LINE__ to kernel function is going to give a mesaurable performance penalty is really hilarious however :) It is crystal clear you really don't understand how to make reliable benchmarks (and likely you don't really have a grasp of nowaday's machine contention points), so why you keep talking about it? It would be more valuable for you and whatever project you follow if you spend your time coding and making real benchmarking. Attilio --=20 Peace can only be achieved by understanding - A. Einstein From owner-freebsd-performance@FreeBSD.ORG Fri Dec 16 16:43:30 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 059511065678; Fri, 16 Dec 2011 16:43:30 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6D9898FC25; Fri, 16 Dec 2011 16:43:28 +0000 (UTC) Received: by vcbfk1 with SMTP id fk1so4457132vcb.13 for ; Fri, 16 Dec 2011 08:43:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=XEYl0wTkXG1IL4Pjffk3B4QbOIqIq8MwBXCjEJejHas=; b=t5UL9rRd2wH80a7awhSuAQtR2gPpgGt1YkLMuQo1cX4ZODtkP4Gbp2zmZklJWH7gHR ygKGdYq/kK6LQPXHL4abExs7pz/NAprYwGQcM1HXwN5G84wllrAdsqGcfbLGiqiJ1M7E u294CtOfDrASMI8oa0OvTjStJCS+qVjagzXvU= MIME-Version: 1.0 Received: by 10.220.151.204 with SMTP id d12mr4077535vcw.40.1324053808493; Fri, 16 Dec 2011 08:43:28 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.52.26.50 with HTTP; Fri, 16 Dec 2011 08:43:27 -0800 (PST) In-Reply-To: <4EEB42B1.1000506@freebsd.org> References: <4EE1EAFE.3070408@m5p.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EEAE8DF.40303@rewt.org.uk> <4EEAEDE1.50604@zedat.fu-berlin.de> <4EEB42B1.1000506@freebsd.org> Date: Fri, 16 Dec 2011 08:43:27 -0800 X-Google-Sender-Auth: KjC5an4W3kNycKQhm4CA4y8apPY Message-ID: From: Adrian Chadd To: Stefan Esser Content-Type: text/plain; charset=ISO-8859-1 Cc: Joe Holden , FreeBSD Stable Mailing List , Current FreeBSD , Arnaud Lacombe , freebsd-performance@freebsd.org, "O. Hartmann" , Jeremy Chadwick Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 16:43:30 -0000 Can someone please write up a nice, concise blog post somewhere outlining all of this? Extra bonus points if it's a blog that is picked up by blogs.freebsdish.org and/or some of the other BSD sites. Guys/girls/fuzzy things - this is 2011; people look at shiny blog sites with graphs rather than mailing lists. Sorry, we lost that battle. :) Adrian From owner-freebsd-performance@FreeBSD.ORG Sat Dec 17 13:26:45 2011 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F6A4106566C for ; Sat, 17 Dec 2011 13:26:45 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [IPv6:2a01:4f8:150:6101::4]) by mx1.freebsd.org (Postfix) with ESMTP id E0E728FC0C for ; Sat, 17 Dec 2011 13:26:41 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id 0F7D1E63B for ; Sat, 17 Dec 2011 14:26:41 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core.vx.sk (amavisd-new, unix socket) with LMTP id HL6oPzlwqwR4 for ; Sat, 17 Dec 2011 14:26:38 +0100 (CET) Received: from [10.9.8.1] (188-167-78-15.dynamic.chello.sk [188.167.78.15]) by mail.vx.sk (Postfix) with ESMTPSA id B4CFBE62C for ; Sat, 17 Dec 2011 14:26:38 +0100 (CET) Message-ID: <4EEC9890.7040806@FreeBSD.org> Date: Sat, 17 Dec 2011 14:26:40 +0100 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: freebsd-performance@FreeBSD.org X-Enigmail-Version: 1.3.4 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Subject: PostgreSQL user experience: FreeBSD (ZFS) vs OpenIndiana (ZFS) vs Linux (EXT4) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 13:26:45 -0000 Hi everyone, I would like to share some of our expreience with PostgreSQL on FreeBSD. It has been a while ago since we had to stop using FreeBSD for our customer's PostgreSQL servers. PostgreSQL (8.4 and 9.0) was demonstrating slow performance under heavy loads and one day I decided to compare it to other alternatives on the very same system (one 8-core with 16GB RAM, another 12-core with 48GB RAM). With ZFS it was extremely slow and with UFS the speed was acceptable (still 10-20% slower) if not under high load. Our databases have a size from several gigabytes to tens of gigabytes. A single real-world query on a idle system was noticeably slower on a FreeBSD system than on the other systems (ZFS and UFS). We compared the EXPLAIN ANALYZE output and the performance penalty was almost equally spread on all items. With rising loads, PostgreSQL processes remain a long time in "semwait" and "msgwait" states and the "top" output shows a high system load on FreeBSD. I have also tried different tunings, compilers and optimizations, but with that I was able to gain only 5-10% better results. The result of a pgbench run by one of my customers on a 12-core system with 48GB RAM is here (FreeBSD ZFS vs OpenIndiana (ZFS) vs Linux (EXT4): http://www.vx.sk/benchmarks/postgresql/pgbench_20110630.ods So our decision so far is the following: - if we are building a PostgreSQL server for heavy loads, we prefer Solaris/OpenIndiana (ZFS or UFS) or Linux (EXT4) to FreeBSD (ZFS or UFS) - if we want to use PostgreSQL on FreeBSD, we prefer UFS to ZFS P.S: Our webservers still run FreeBSD and e.g. OpenIndiana (Solaris) performed much worse in our high load real-world web application. -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-performance@FreeBSD.ORG Sat Dec 17 15:25:43 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04A2E1065670; Sat, 17 Dec 2011 15:25:43 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 610E78FC13; Sat, 17 Dec 2011 15:25:42 +0000 (UTC) Received: by wgbds13 with SMTP id ds13so4794938wgb.1 for ; Sat, 17 Dec 2011 07:25:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=nTbyBdzJWT2Qwt/qaMF3tXrKYw9o7fE/LlrR5k4ZW0E=; b=oxmv7q+UsG3BLkSpzjtpdoDJg91iHh7ZkGkpC6yETjKJBG6mGK3GXKQgG8SdDsq0RT xnWyuaC9qfArey2EYSuN0FjFDCa0wJEgym3ptmU9PHG02VH8B/amCBx6oNoadbHsm0pM dABXJSWr2bLL76984Hyui8wzNIKqEl58sPvGI= MIME-Version: 1.0 Received: by 10.180.82.138 with SMTP id i10mr4497808wiy.2.1324133980360; Sat, 17 Dec 2011 06:59:40 -0800 (PST) Received: by 10.223.156.132 with HTTP; Sat, 17 Dec 2011 06:59:40 -0800 (PST) In-Reply-To: <4EEC9890.7040806@FreeBSD.org> References: <4EEC9890.7040806@FreeBSD.org> Date: Sat, 17 Dec 2011 08:59:40 -0600 Message-ID: From: Adam Vande More To: Martin Matuska Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: PostgreSQL user experience: FreeBSD (ZFS) vs OpenIndiana (ZFS) vs Linux (EXT4) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Dec 2011 15:25:43 -0000 On Sat, Dec 17, 2011 at 7:26 AM, Martin Matuska wrote: > I would like to share some of our expreience with PostgreSQL on FreeBSD. > It has been a while ago since we had to stop using FreeBSD for our > customer's PostgreSQL servers. > I don't claim any expertise in this area so please take this with that in mind. A couple of things that might help. Setting zfs set sync=disabled on fs where the WAL resides(all alone). Of course this would help ZFS regardless of OS. http://www.postgresql.org/docs/8.4/static/wal-intro.html I've also seen other reports of slow PostgreSQL on FreeBSD, and in that scenario there were a large amount of gettimeofday(2) calls as the bottleneck and such calls on FreeBSD are significantly slower. http://postgresql.1045698.n5.nabble.com/Postgresql-9-0-2-explain-analyze-very-slow-10x-compared-to-actual-query-time-td3336664.html If you have a faster timecounter, it might help to switch. -- Adam Vande More