From owner-freebsd-performance@FreeBSD.ORG Thu Jul 7 05:55:43 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4AB781065674; Thu, 7 Jul 2011 05:55:43 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 026338FC0A; Thu, 7 Jul 2011 05:55:42 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1QehYs-00014g-2g>; Thu, 07 Jul 2011 07:55:42 +0200 Received: from e178008197.adsl.alicedsl.de ([85.178.8.197] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1QehYr-000276-VO>; Thu, 07 Jul 2011 07:55:42 +0200 Message-ID: <4E154A5D.8080009@zedat.fu-berlin.de> Date: Thu, 07 Jul 2011 07:55:41 +0200 From: "Hartmann, O." User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110630 Thunderbird/5.0 MIME-Version: 1.0 To: Arnaud Lacombe , FreeBSD Current , "freebsd-performance@freebsd.org" References: <4E1421D9.7080808@zedat.fu-berlin.de> <4E147F54.40908@zedat.fu-berlin.de> <20110706162811.GA68436@troutmask.apl.washington.edu> <20110706193636.GA69550@troutmask.apl.washington.edu> <4E14CCE5.4050906@zedat.fu-berlin.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: 85.178.8.197 Cc: Subject: Re: Heavy I/O blocks FreeBSD box for several seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 05:55:43 -0000 On 07/07/11 06:29, Arnaud Lacombe wrote: > Hi, > > On Wed, Jul 6, 2011 at 5:00 PM, Hartmann, O. > wrote: >> On 07/06/11 21:36, Steve Kargl wrote: >>> On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote: >>>> Hi, >>>> >>>> On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl >>>> wrote: >>>>> On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote: >>>>>> I use SCHED_ULE on all machines, since it is supposed to be performing >>>>>> better on multicore boxes, but there are lots of suggestions switching >>>>>> back to the old SCHED_4BSD scheduler. >>>>>> >>>>> If you are using MPI in numerical codes, then you want >>>>> to use SCHED_4BSD. ?I've posted numerous times about ULE >>>>> and its very poor performance when using MPI. >>>>> >>>>> >>>>> http://lists.freebsd.org/pipermail/freebsd-hackers/2008-October/026375.html >>>>> >>>> [sarcasm] >>>> It is rather funny to see that the post you point out has generated >>>> exactly 0 meaningful follow-up then and as you mention later in this >>>> thread, the issue still remains today :-) >>>> [/sarcasm] >>>> >>> Apparently, you are privy to my private email exchanges >>> with jeffr. >>> >>> I'm also not sure why you're being sarcastic here. The >>> issue was and AFAIK still is a problem for anyone using >>> FreeBSD in a HPC cluster. ULE simply performs worse than >>> 4BSD. >>> >> Well, I know only very little people using FreeBSD within a HPC cluster or >> even for scientific purposes, except myself and some people around here. >> > Well, quad-core CPU, dual-socket machine are quite common these day, > even in non-HPC system. So, unless you understand enough the issue and > ULE to assert that this issue is tight to this workload only, I would > assume this issue to affect other ULE use-case and a broader user > spectrum than "you and some people around". > > - Arnaud Maybe this is a little misunderstanding. I complained about the fact that FreeBSD is more and more vanishing from HPC (it was very common in the mid 90s and at the beginning the 2000s). My former department banned after the introduction of Linux kernel 2.6 all FreeBSD boxes due to a much better performance (network) and the availability of HPC 64bit compilers. Nevertheless, nowadays the situation has turned even worse with GPGPU. As you said, multicores are very common and so the inabilities of the multicore-aware scheduler ULE become effective not even for a marginal group of users with a specific workload. From owner-freebsd-performance@FreeBSD.ORG Thu Jul 7 08:20:30 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A55DB106564A for ; Thu, 7 Jul 2011 08:20:30 +0000 (UTC) (envelope-from case.vanrij@gmail.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id 604538FC15 for ; Thu, 7 Jul 2011 08:20:30 +0000 (UTC) Received: by yic13 with SMTP id 13so364234yic.13 for ; Thu, 07 Jul 2011 01:20:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=p0D56lVfkElHKkrTW47sALAUTuW2yWCLJHDv8aPOgkw=; b=A6YzL7Yw+OuwKg+vBZhhjD5kkOd8IhwqAuM+Q1zBXBiHSUlFp3Wq60W1bDY2hu0F/N v1B0JAoAPcY7SWFV3Ug8/Sp4hzWGKmVd6HMTmY0tutXCQeTIynRTYO7ES8vh5/N7aOqR GrcwIilF1c9JlpmNNzzKcBkJvADOF4l7LuOtQ= MIME-Version: 1.0 Received: by 10.236.144.231 with SMTP id n67mr519270yhj.354.1310025337524; Thu, 07 Jul 2011 00:55:37 -0700 (PDT) Received: by 10.236.106.65 with HTTP; Thu, 7 Jul 2011 00:55:37 -0700 (PDT) In-Reply-To: <4E154A5D.8080009@zedat.fu-berlin.de> References: <4E1421D9.7080808@zedat.fu-berlin.de> <4E147F54.40908@zedat.fu-berlin.de> <20110706162811.GA68436@troutmask.apl.washington.edu> <20110706193636.GA69550@troutmask.apl.washington.edu> <4E14CCE5.4050906@zedat.fu-berlin.de> <4E154A5D.8080009@zedat.fu-berlin.de> Date: Thu, 7 Jul 2011 00:55:37 -0700 Message-ID: From: Case van Rij To: sgk@troutmask.apl.washington.edu Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-performance@freebsd.org" , Arnaud Lacombe Subject: Re: Heavy I/O blocks FreeBSD box for several seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 08:20:30 -0000 On Wed, Jul 6, 2011 at 10:55 PM, Hartmann, O. wrote: > On 07/07/11 06:29, Arnaud Lacombe wrote: >> >> Hi, >> >> On Wed, Jul 6, 2011 at 5:00 PM, Hartmann, O. >> =A0wrote: >>> >>> On 07/06/11 21:36, Steve Kargl wrote: >>>> >>>> On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote: >>>>> >>>>> Hi, >>>>> >>>>> On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl >>>>> =A0 =A0wrote: >>>>>> >>>>>> On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote: >>>>>>> >>>>>>> I use SCHED_ULE on all machines, since it is supposed to be >>>>>>> performing >>>>>>> better on multicore boxes, but there are lots of suggestions >>>>>>> switching >>>>>>> back to the old SCHED_4BSD scheduler. >>>>>>> >>>>>> If you are using MPI in numerical codes, then you want >>>>>> to use SCHED_4BSD. ?I've posted numerous times about ULE >>>>>> and its very poor performance when using MPI. >>>>>> >>>>>> http://lists.freebsd.org/pipermail/freebsd-hackers/2008-October/0263= 75.html >>>>>>>With ULE, 2 Test_mpi jobs are always scheduled on the same core whil= e one >>>>>>>core remains idle. Also, note the difference in the reported load a= verages. While possibly not the same issue you're seeing, I noticed a similar problem on 8 and 12 core machines with ULE, specifically with a relatively small number of threads runnable but waiting to run on a busy core while other cores were sitting idle. tdq_idled won't steal threads from a queue unless there are kern.sched.steal_thresh threads in that queue, where steal_thresh =3D min(fls(mp_ncpus) - 1, 3); ie. on an 8 core system you need 3 threads in the queue before idled steals one. Fortunately you can simply override steal_thresh at run time. 1 works great for me, ymmv. From owner-freebsd-performance@FreeBSD.ORG Thu Jul 7 22:51:04 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4850C106564A; Thu, 7 Jul 2011 22:51:04 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id EC6AB8FC14; Thu, 7 Jul 2011 22:51:03 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1QexPS-0003Jx-Pk>; Fri, 08 Jul 2011 00:51:02 +0200 Received: from e178023242.adsl.alicedsl.de ([85.178.23.242] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1QexPS-0002J5-MI>; Fri, 08 Jul 2011 00:51:02 +0200 Message-ID: <4E163856.2050300@zedat.fu-berlin.de> Date: Fri, 08 Jul 2011 00:51:02 +0200 From: "Hartmann, O." User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110630 Thunderbird/5.0 MIME-Version: 1.0 To: Andriy Gapon References: <4E1421D9.7080808@zedat.fu-berlin.de> <4E147F54.40908@zedat.fu-berlin.de> <20110706162811.GA68436@troutmask.apl.washington.edu> <20110706193636.GA69550@troutmask.apl.washington.edu> <4E14CCE5.4050906@zedat.fu-berlin.de> <20110707015151.GB71966@troutmask.apl.washington.edu> <20110707031151.GA72452@troutmask.apl.washington.edu> <4E155A84.1010100@FreeBSD.org> In-Reply-To: <4E155A84.1010100@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: 85.178.23.242 Cc: Adrian Chadd , Steve Kargl , arrowdodger <6yearold@gmail.com>, FreeBSD Current , Arnaud Lacombe , "freebsd-performance@freebsd.org" Subject: Re: Heavy I/O blocks FreeBSD box for several seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 22:51:04 -0000 On 07/07/11 09:04, Andriy Gapon wrote: > on 07/07/2011 06:11 Steve Kargl said the following: >> Unfortunately, I have neither the brain capacity and time nor >> the money to fix the issue. To solve OP's problem in the >> short, the simplest solution may be to switch to 4BSD. Let's >> face, ULE is not a silver bullet. > I think that piling up different problems into a single discussion, even if they > involve a common component (to a certain degree), is not going to help anybody. > If I have read this thread correctly (and taking the subject line as a witness) > the OP had a problem with heavy I/O activity screwing up interactivity. I think > that it's not the same problem as sub-optimal performance of heavy CPU-bound > load, which is what you reported if I am not mistaken. > This is quibbling. On heavy loads on networ, disk et cetera, isn't there always and also a CPU bound load? Whenever this problem came up, it was brought down by force. Yes, I reported due to the obvious fact that this essential problem involves usability of several workstations. It get more obvious when FreeBSD is used with a GUI. But I also realized, as I reported(!), problems on a headless server, even with several tunings, performed slowly and with increasing numbers over time as recommended here (for instance, kern.sched.preempt_thresh=224 or up to kern.sched.preempt_thresh=512, with little effect). Where, if not here, should such problems be discussed? If we open for each dedicated micro-problem a separate thread, we would never gather that many problems which seem to be related to one single well known 'sweet spot'. From owner-freebsd-performance@FreeBSD.ORG Thu Jul 7 23:13:43 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD45E106566C; Thu, 7 Jul 2011 23:13:43 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 646018FC0A; Thu, 7 Jul 2011 23:13:42 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1QexlN-0005TK-Eg>; Fri, 08 Jul 2011 01:13:41 +0200 Received: from e178023242.adsl.alicedsl.de ([85.178.23.242] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1QexlN-0003Oa-B4>; Fri, 08 Jul 2011 01:13:41 +0200 Message-ID: <4E163DA4.6060505@zedat.fu-berlin.de> Date: Fri, 08 Jul 2011 01:13:40 +0200 From: "Hartmann, O." User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110630 Thunderbird/5.0 MIME-Version: 1.0 To: Andriy Gapon References: <20110706170132.GA68775@troutmask.apl.washington.edu> <5080.1309971941@critter.freebsd.dk> <20110706180001.GA69157@troutmask.apl.washington.edu> <4E14A54A.4050106@freebsd.org> <4E155FF9.5090905@FreeBSD.org> In-Reply-To: <4E155FF9.5090905@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: 85.178.23.242 Cc: "freebsd-performance@freebsd.org" , FreeBSD Current , Nathan Whitehorn , Steve Kargl Subject: Re: Heavy I/O blocks FreeBSD box for several seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 23:13:43 -0000 On 07/07/11 09:27, Andriy Gapon wrote: > on 06/07/2011 21:11 Nathan Whitehorn said the following: >> On 07/06/11 13:00, Steve Kargl wrote: >>> AFAICT, it is a cpu affinity issue. If I launch n+1 MPI images >>> on a system with n cpus/cores, then 2 (and sometimes 3) images >>> are stuck on a cpu and those 2 (or 3) images ping-pong on that >>> cpu. I recall trying to use renice(8) to force some load >>> balancing, but vaguely remember that it did not help. >> I've seen exactly this problem with multi-threaded math libraries, as well. > Exactly the same? Let's see. > >> Using parallel GotoBLAS on FreeBSD gives terrible performance because the >> threads keep migrating between CPUs, causing frequent cache misses. > So Steve reports that if he has Nthr> Ncpu, then some threads are "over-glued" > to a particular CPU, which results in sub-optimal scheduling for those threads. > I have to guess that Steve would want to see the threads being shuffled between > CPUs to produce more even CPU load. > > On the other hand, you report that your threads keep being shuffled between CPUs > (I presume for Nthr == Ncpu case, where Nthr is a count of the number-crunching > threads). And I guess that you want them to stay glued to particular CPUs. > > So how is this the same problem? In fact, it sounds like somewhat opposite. > The only thing in common is that you both don't like how ULE works. > > ULE has many knobs to tune its behavior. Unfortunately they are not very well > documented and there are too many of them. So, it's not easy to find which > combination would be the best for a particular work-load. In your particular > case you might want to try to increase value of kern.sched.affinity to increase > affinity of threads to their CPUs. Not all of those using FreeBSD are developer or experts, even experts of a very specific area of computer science and engineering or a particular subject of the FreeBSD kernel and its techniques of scheduling. I'm not capable of tuning my servers via a lot of undocumented knobs, I'm sorry. I'd like to do if there would be a kind of howto (handbook?). > > Also, please note that FreeBSD support in GotoBLAS is not equivalent to Linux > support as I have pointed out before. On Linux they bind their threads to CPUs > to avoid the situation that you describe. Apparently they didn't know how to do > CPU-binding on FreeBSD, so this is not implemented. You may have a motivation > to help them out with this. > From owner-freebsd-performance@FreeBSD.ORG Fri Jul 8 00:19:10 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B452D106564A for ; Fri, 8 Jul 2011 00:19:10 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from asmtpout029.mac.com (asmtpout029.mac.com [17.148.16.104]) by mx1.freebsd.org (Postfix) with ESMTP id 921D18FC1D for ; Fri, 8 Jul 2011 00:19:10 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from cswiger1.apple.com ([17.209.4.71]) by asmtp029.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01 64bit (built Nov 21 2010)) with ESMTPSA id <0LNZ00M56KMIYE60@asmtp029.mac.com>; Thu, 07 Jul 2011 16:15:55 -0700 (PDT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.4.6813,1.0.211,0.0.0000 definitions=2011-07-07_10:2011-07-07, 2011-07-07, 1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1107070220 From: Chuck Swiger In-reply-to: <4E163856.2050300@zedat.fu-berlin.de> Date: Thu, 07 Jul 2011 16:15:54 -0700 Message-id: <5C3302CF-8EE7-489E-8D31-624DA1637B73@mac.com> References: <4E1421D9.7080808@zedat.fu-berlin.de> <4E147F54.40908@zedat.fu-berlin.de> <20110706162811.GA68436@troutmask.apl.washington.edu> <20110706193636.GA69550@troutmask.apl.washington.edu> <4E14CCE5.4050906@zedat.fu-berlin.de> <20110707015151.GB71966@troutmask.apl.washington.edu> <20110707031151.GA72452@troutmask.apl.washington.edu> <4E155A84.1010100@FreeBSD.org> <4E163856.2050300@zedat.fu-berlin.de> To: "Hartmann, O." X-Mailer: Apple Mail (2.1084) Cc: freebsd-performance@freebsd.org, FreeBSD current Subject: Re: Heavy I/O blocks FreeBSD box for several seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2011 00:19:10 -0000 On Jul 7, 2011, at 3:51 PM, Hartmann, O. wrote: > This is quibbling. On heavy loads on networ, disk et cetera, isn't there always and also a CPU bound load? No. Properly written software blocks when waiting on network or disk I/O, and doesn't sit there spinning in a busy-wait consuming CPU until it actually gets more work to do. See select(2), kqueue(2), and friends. Regards, -- -Chuck