From owner-freebsd-smp Sun Apr 28 3:15: 3 2002 Delivered-To: freebsd-smp@freebsd.org Received: from rina.r.dl.itc.u-tokyo.ac.jp (cvsup2.r.dl.itc.u-tokyo.ac.jp [133.11.199.247]) by hub.freebsd.org (Postfix) with ESMTP id 1DA4237B404; Sun, 28 Apr 2002 03:14:59 -0700 (PDT) Received: from silver.carrots.uucp.r.dl.itc.u-tokyo.ac.jp (silver.carrots.uucp.r.dl.itc.u-tokyo.ac.jp [IPv6:3ffe:b80:5b0:3:280:c8ff:fe6b:6d73]) by rina.r.dl.itc.u-tokyo.ac.jp (8.12.2/3.7W-rina.r-Nankai-Koya) with ESMTP id g3SAEu0o012772 ; Sun, 28 Apr 2002 19:14:57 +0900 (JST) Received: from silver.carrots.uucp.r.dl.itc.u-tokyo.ac.jp (localhost [127.0.0.1]) by silver.carrots.uucp.r.dl.itc.u-tokyo.ac.jp (8.12.2/3.7W-carrots-Keikyu-Kurihama) with ESMTP id g3SADEPw057058 ; Sun, 28 Apr 2002 19:13:34 +0900 (JST) Message-Id: <200204281013.g3SADEPw057058@silver.carrots.uucp.r.dl.itc.u-tokyo.ac.jp> Date: Sun, 28 Apr 2002 19:13:13 +0900 From: Seigo Tanimura To: current@FreeBSD.org, smp@FreeBSD.org Subject: Re: Locking down a socket, milestone 1 In-Reply-To: <200204241110.g3OB8u8t006194@bunko> References: <200204241110.g3OB8u8t006194@bunko> Cc: Seigo Tanimura User-Agent: Wanderlust/2.8.1 (Something) SEMI/1.14.3 (Ushinoya) FLIM/1.14.3 (=?ISO-8859-1?Q?Unebigory=F2mae?=) APEL/10.3 MULE XEmacs/21.1 (patch 14) (Cuyahoga Valley) (i386--freebsd) Organization: Digital Library Research Division, Information Techinology Centre, The University of Tokyo MIME-Version: 1.0 (generated by SEMI 1.14.3 - "Ushinoya") Content-Type: text/plain; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Wed, 24 Apr 2002 20:08:56 +0900, Seigo Tanimura said: Seigo> I am now working on locking down a socket. (I have heard that Jeffrey Seigo> Hsu is also doing that, but I have never seen his patch. Has anyone Seigo> seen that?) My first milestone patch is now available at: Seigo> http://people.FreeBSD.org/~tanimura/patches/socket_milestone1.diff.gz I ripped off and committed the part of a sigio lock. The updated patch is found at: http://people.FreeBSD.org/~tanimura/patches/socket_milestone1a.diff.gz -- Seigo Tanimura To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Apr 28 4:22:34 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id EFE9937B41A; Sun, 28 Apr 2002 04:22:29 -0700 (PDT) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.9.3/8.8.7) with ESMTP id VAA29629; Sun, 28 Apr 2002 21:22:18 +1000 Date: Sun, 28 Apr 2002 21:23:33 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Seigo Tanimura Cc: current@FreeBSD.ORG, Subject: Re: Locking down a socket, milestone 1 In-Reply-To: <200204281013.g3SADEPw057058@silver.carrots.uucp.r.dl.itc.u-tokyo.ac.jp> Message-ID: <20020428212122.W3806-100000@gamplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Sun, 28 Apr 2002, Seigo Tanimura wrote: > On Wed, 24 Apr 2002 20:08:56 +0900, > Seigo Tanimura said: > > Seigo> I am now working on locking down a socket. (I have heard that Jeffrey > Seigo> Hsu is also doing that, but I have never seen his patch. Has anyone > Seigo> seen that?) My first milestone patch is now available at: > > > Seigo> http://people.FreeBSD.org/~tanimura/patches/socket_milestone1.diff.gz > > I ripped off and committed the part of a sigio lock. The updated > patch is found at: The committed part re-adds lots of namespace pollution to and . Please fix this. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 29 13:49:49 2002 Delivered-To: freebsd-smp@freebsd.org Received: from BYHU.star2.net (secure.star2.net [65.89.75.12]) by hub.freebsd.org (Postfix) with ESMTP id CB05C37B405 for ; Mon, 29 Apr 2002 13:49:47 -0700 (PDT) Received: from cb804915a (unverified [63.14.88.238]) by BYHU.star2.net (Vircom SMTPRS 5.1.200) with SMTP id for ; Mon, 29 Apr 2002 16:14:29 -0400 Reply-To: From: "Alan Matykiewicz" To: Subject: subscribe Date: Mon, 29 Apr 2002 16:14:39 -0400 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Subscribe To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 6:54:23 2002 Delivered-To: freebsd-smp@freebsd.org Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by hub.freebsd.org (Postfix) with ESMTP id 5629337B416 for ; Wed, 1 May 2002 06:54:20 -0700 (PDT) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id JAA02059 for ; Wed, 1 May 2002 09:54:19 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id g41DrnD18153; Wed, 1 May 2002 09:53:49 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15567.62317.677224.3470@grasshopper.cs.duke.edu> Date: Wed, 1 May 2002 09:53:49 -0400 (EDT) To: freebsd-smp@freebsd.org Subject: hlt when idle? X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Can somebody remind me why we do not hlt in the idle loop on MP x86s? Is this because a HLTed CPU is not going to notice a new runnable job (possibly migrating from another CPU) until it gets an interrupt to wake it up? Do both CPUs get clock interrupts on x86? Anyway, am I able to remove the SMP ifdef's from the default_halt function (on 4-stable, 2x1GHz PIII) and I have not noticed any change other than a cooler office & longer runtime on a UPS. (I've not timed a buildworld). Thanks, Drew To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 10:16: 0 2002 Delivered-To: freebsd-smp@freebsd.org Received: from gull.prod.itd.earthlink.net (gull.mail.pas.earthlink.net [207.217.120.84]) by hub.freebsd.org (Postfix) with ESMTP id E4E5337B419 for ; Wed, 1 May 2002 10:15:49 -0700 (PDT) Received: from pool0332.cvx40-bradley.dialup.earthlink.net ([216.244.43.77] helo=mindspring.com) by gull.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 172xhi-0004Ou-00; Wed, 01 May 2002 10:15:47 -0700 Message-ID: <3CD022A2.FACD21E9@mindspring.com> Date: Wed, 01 May 2002 10:15:14 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Andrew Gallatin Cc: freebsd-smp@freebsd.org Subject: Re: hlt when idle? References: <15567.62317.677224.3470@grasshopper.cs.duke.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Andrew Gallatin wrote: > Can somebody remind me why we do not hlt in the idle loop on MP x86s? Halting with "giant" and/or "the scheduler lock" held is bad. > Is this because a HLTed CPU is not going to notice a new runnable job > (possibly migrating from another CPU) until it gets an interrupt to > wake it up? Partly. It's worse than that, if the system is relatively quiescent. Interrupts are routed with entry to the kernel, the system doesn't run in "virtual wire" mode, and it doesn't support seperately routing interrupts. What this means is that it's possible for an intertrupt event on the non-halted processor to result in work for the halted processor being available, and the halted processor never noticing it. The net effect is that one CPU goes to sleep and stays there, either for a long time, or indefinitely, depending on the timing and load at the time. > Do both CPUs get clock interrupts on x86? The clock interrupts may or may not be routed via the APIC; routing of the clock interrupt is one of the most common bogosities that you have to handle on x86 SMP systems; there is a special printf for complaining about having to work around broken BIOS and/or motherboard circuitry. In theory, you want to run in "virtual wire mode", which means that the CPUs get the interrupt, and then contend in order to see who services it. Another way to handle this would be to manually send IPIs for certain events, like jobs being at the head of the scheduler queue. For this to work right, it's probably be a good idea to record when a CPU is going idle, so that it can be targetted specifically for wakeup, rather than targetting everyone. It's ugly. > Anyway, am I able to remove the SMP ifdef's from the default_halt > function (on 4-stable, 2x1GHz PIII) and I have not noticed any change other > than a cooler office & longer runtime on a UPS. (I've not timed a > buildworld). You might be lucky in your SMP hardware, and you might just not be noticing the drop out of one CPU. 8-(. Actually, the whole SMP HLT thing comes up once every six months or so. There's a much better discussion on the whole issue, and the negative effects of the HLT, which took place on the SMP list about 8 moths ago, and again, about 12 months ago. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 10:48:10 2002 Delivered-To: freebsd-smp@freebsd.org Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by hub.freebsd.org (Postfix) with ESMTP id F32A937B416 for ; Wed, 1 May 2002 10:48:02 -0700 (PDT) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id NAA10648; Wed, 1 May 2002 13:48:02 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id g41HlW218398; Wed, 1 May 2002 13:47:32 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15568.10804.364425.409745@grasshopper.cs.duke.edu> Date: Wed, 1 May 2002 13:47:32 -0400 (EDT) To: Terry Lambert Cc: freebsd-smp@freebsd.org Subject: Re: hlt when idle? In-Reply-To: <3CD022A2.FACD21E9@mindspring.com> References: <15567.62317.677224.3470@grasshopper.cs.duke.edu> <3CD022A2.FACD21E9@mindspring.com> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Terry Lambert writes: > Andrew Gallatin wrote: > > Can somebody remind me why we do not hlt in the idle loop on MP x86s? > > Halting with "giant" and/or "the scheduler lock" held is bad. Ug. We hold locks in the idle loop?? > > Do both CPUs get clock interrupts on x86? > > The clock interrupts may or may not be routed via the APIC; > routing of the clock interrupt is one of the most common > bogosities that you have to handle on x86 SMP systems; there > is a special printf for complaining about having to work > around broken BIOS and/or motherboard circuitry. Is this what you mean: APIC_IO: Testing 8254 interrupt delivery APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin 2 APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 This was taken from the machine which seems to work just fine with halting in the idle loop... (Serverworks LE based SuperMicro 370DLE) Drew To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 12:34:55 2002 Delivered-To: freebsd-smp@freebsd.org Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by hub.freebsd.org (Postfix) with ESMTP id C1E2737B405 for ; Wed, 1 May 2002 12:34:39 -0700 (PDT) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id PAA14412; Wed, 1 May 2002 15:34:39 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id g41JY9626047; Wed, 1 May 2002 15:34:09 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15568.17201.80743.521864@grasshopper.cs.duke.edu> Date: Wed, 1 May 2002 15:34:09 -0400 (EDT) To: Terry Lambert Cc: freebsd-smp@freebsd.org Subject: Re: hlt when idle? In-Reply-To: <3CD022A2.FACD21E9@mindspring.com> References: <15567.62317.677224.3470@grasshopper.cs.duke.edu> <3CD022A2.FACD21E9@mindspring.com> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Terry Lambert writes: > > Anyway, am I able to remove the SMP ifdef's from the default_halt > > function (on 4-stable, 2x1GHz PIII) and I have not noticed any change other > > than a cooler office & longer runtime on a UPS. (I've not timed a > > buildworld). > > You might be lucky in your SMP hardware, and you might just not > be noticing the drop out of one CPU. 8-(. FWIW, I just did a -j8 buildworld with & without HLT. The times were nearly identical, with the HLT kernel being ~15 seconds faster (real time) and using 56 seconds less system time. With a sample size of 1, I admit its hard to make any meaningful performance comparisons; but the system seems roughly as fast & did not deadlock. At least on these machines, the cooler office seems worth the risk ;) Drew ==> world.log.hlt <== 1757.06 real 1694.33 user 674.49 sys 16672 maximum resident set size 1060 average shared memory size 1059 average unshared data size 129 average unshared stack size 11280502 page reclaims 2380 page faults 0 swaps 24326 block input operations 4162 block output operations 18 messages sent 18 messages received 8 signals received 588857 voluntary context switches 371028 involuntary context switches ==> world.log <== 1771.65 real 1696.62 user 730.04 sys 16668 maximum resident set size 1049 average shared memory size 1043 average unshared data size 129 average unshared stack size 11268611 page reclaims 1901 page faults 0 swaps 17717 block input operations 4445 block output operations 19 messages sent 19 messages received 8 signals received 572960 voluntary context switches 363564 involuntary context switches To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 12:46:58 2002 Delivered-To: freebsd-smp@freebsd.org Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by hub.freebsd.org (Postfix) with ESMTP id DBD8237B404 for ; Wed, 1 May 2002 12:46:54 -0700 (PDT) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.2/8.12.2) with ESMTP id g41JkPqE068723; Wed, 1 May 2002 21:46:26 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Andrew Gallatin Cc: freebsd-smp@FreeBSD.ORG Subject: Re: hlt when idle? In-Reply-To: Your message of "Wed, 01 May 2002 15:34:09 EDT." <15568.17201.80743.521864@grasshopper.cs.duke.edu> Date: Wed, 01 May 2002 21:46:25 +0200 Message-ID: <68722.1020282385@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org In message <15568.17201.80743.521864@grasshopper.cs.duke.edu>, Andrew Gallatin writes: >FWIW, I just did a -j8 buildworld with & without HLT. The times were >nearly identical, with the HLT kernel being ~15 seconds faster (real >time) and using 56 seconds less system time. With a sample size of 1, >I admit its hard to make any meaningful performance comparisons; >but the system seems roughly as fast & did not deadlock. > >At least on these machines, the cooler office seems worth the risk ;) > >Drew > >==> world.log.hlt <== > 1757.06 real 1694.33 user 674.49 sys > >==> world.log <== > 1771.65 real 1696.62 user 730.04 sys Your system time looks significant, the other two are a bit too close to judge without a standard deviation. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 13:16:37 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail.speakeasy.net (mail13.speakeasy.net [216.254.0.213]) by hub.freebsd.org (Postfix) with ESMTP id C39A037B421 for ; Wed, 1 May 2002 13:16:33 -0700 (PDT) Received: (qmail 10642 invoked from network); 1 May 2002 20:16:31 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail13.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 1 May 2002 20:16:31 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g41KGSF04381; Wed, 1 May 2002 16:16:28 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <15567.62317.677224.3470@grasshopper.cs.duke.edu> Date: Wed, 01 May 2002 16:15:28 -0400 (EDT) From: John Baldwin To: Andrew Gallatin Subject: RE: hlt when idle? Cc: freebsd-smp@freebsd.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 01-May-2002 Andrew Gallatin wrote: > > > Can somebody remind me why we do not hlt in the idle loop on MP x86s? > Is this because a HLTed CPU is not going to notice a new runnable job > (possibly migrating from another CPU) until it gets an interrupt to > wake it up? Yes. > Do both CPUs get clock interrupts on x86? No, the interrupts seem to be round-robin, but each clock intr is only sent to one CPU unlike on alpha where they are broadcast. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 13:16:55 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail.speakeasy.net (mail17.speakeasy.net [216.254.0.217]) by hub.freebsd.org (Postfix) with ESMTP id EDD5F37B404 for ; Wed, 1 May 2002 13:16:37 -0700 (PDT) Received: (qmail 3382 invoked from network); 1 May 2002 20:16:36 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail17.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 1 May 2002 20:16:36 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g41KGaF04385; Wed, 1 May 2002 16:16:36 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3CD022A2.FACD21E9@mindspring.com> Date: Wed, 01 May 2002 16:15:35 -0400 (EDT) From: John Baldwin To: Terry Lambert Subject: Re: hlt when idle? Cc: freebsd-smp@freebsd.org, Andrew Gallatin Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 01-May-2002 Terry Lambert wrote: > Andrew Gallatin wrote: >> Can somebody remind me why we do not hlt in the idle loop on MP x86s? > > Halting with "giant" and/or "the scheduler lock" held is bad. Considering the idle loop doesn't hold any locks this doesn't apply. >> Do both CPUs get clock interrupts on x86? > > The clock interrupts may or may not be routed via the APIC; > routing of the clock interrupt is one of the most common > bogosities that you have to handle on x86 SMP systems; there > is a special printf for complaining about having to work > around broken BIOS and/or motherboard circuitry. You don't know what you are talking about. All the interrupts come through the APIC. The hack that we do is to abuse the fact that the normal AT PIC routes through the APIC to piggy back the clock interrupt onto that (via routing it to the AT PIC). However, the interrupt still comes to the CPU from the I/O APIc. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 13:22:55 2002 Delivered-To: freebsd-smp@freebsd.org Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) by hub.freebsd.org (Postfix) with ESMTP id 0BC3737B416; Wed, 1 May 2002 13:22:46 -0700 (PDT) Received: from grasshopper.cs.duke.edu (grasshopper.cs.duke.edu [152.3.145.30]) by duke.cs.duke.edu (8.9.3/8.9.3) with ESMTP id QAA16405; Wed, 1 May 2002 16:22:45 -0400 (EDT) Received: (from gallatin@localhost) by grasshopper.cs.duke.edu (8.11.6/8.9.1) id g41KMFU26818; Wed, 1 May 2002 16:22:15 -0400 (EDT) (envelope-from gallatin@cs.duke.edu) From: Andrew Gallatin MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <15568.20086.979721.992191@grasshopper.cs.duke.edu> Date: Wed, 1 May 2002 16:22:14 -0400 (EDT) To: John Baldwin Cc: freebsd-smp@FreeBSD.org Subject: RE: hlt when idle? In-Reply-To: References: <15567.62317.677224.3470@grasshopper.cs.duke.edu> X-Mailer: VM 6.75 under 21.1 (patch 12) "Channel Islands" XEmacs Lucid Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin writes: > > On 01-May-2002 Andrew Gallatin wrote: > > > > > > Can somebody remind me why we do not hlt in the idle loop on MP x86s? > > Is this because a HLTed CPU is not going to notice a new runnable job > > (possibly migrating from another CPU) until it gets an interrupt to > > wake it up? > > Yes. This seems to be an acceptable "loss" in performance in environments where cooling is a concern. Is there a deadlock danger? Or is it just a performance tweak to not HLT SMPs? Would you object to making it a sysctl (machdep.smp_idle_hlt)? > > Do both CPUs get clock interrupts on x86? > > No, the interrupts seem to be round-robin, but each clock intr is only > sent to one CPU unlike on alpha where they are broadcast. So each CPU gets (1/num_cpu) * hz clock interrupts/sec? Drew To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 15: 6:59 2002 Delivered-To: freebsd-smp@freebsd.org Received: from 12-234-96-171.client.attbi.com (12-234-96-171.client.attbi.com [12.234.96.171]) by hub.freebsd.org (Postfix) with ESMTP id 12D4637B404; Wed, 1 May 2002 15:06:57 -0700 (PDT) Received: by 12-234-96-171.client.attbi.com (Postfix, from userid 1000) id 56D9AA900; Wed, 1 May 2002 15:11:23 -0700 (PDT) Date: Wed, 1 May 2002 15:11:23 -0700 From: Jonathan Mini To: Andrew Gallatin Cc: John Baldwin , freebsd-smp@FreeBSD.ORG Subject: Re: hlt when idle? Message-ID: <20020501151123.G30080@stylus.haikugeek.com> References: <15567.62317.677224.3470@grasshopper.cs.duke.edu> <15568.20086.979721.992191@grasshopper.cs.duke.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <15568.20086.979721.992191@grasshopper.cs.duke.edu>; from gallatin@cs.duke.edu on Wed, May 01, 2002 at 04:22:14PM -0400 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Andrew Gallatin [gallatin@cs.duke.edu] wrote : > > No, the interrupts seem to be round-robin, but each clock intr is only > > sent to one CPU unlike on alpha where they are broadcast. > > So each CPU gets (1/num_cpu) * hz clock interrupts/sec? Yes, but because the timer is set to num_cpu*hz, each CPU ends up getting the normal hz interrupts. That's why it runs round-robin but looks like a broadcast. -- Jonathan Mini http://www.haikugeek.com "He who is not aware of his ignorance will be only misled by his knowledge." -- Richard Whatley To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 18:41:12 2002 Delivered-To: freebsd-smp@freebsd.org Received: from web20908.mail.yahoo.com (web20908.mail.yahoo.com [216.136.226.230]) by hub.freebsd.org (Postfix) with SMTP id 92A2137B404 for ; Wed, 1 May 2002 18:41:08 -0700 (PDT) Message-ID: <20020502014108.24342.qmail@web20908.mail.yahoo.com> Received: from [218.108.155.50] by web20908.mail.yahoo.com via HTTP; Wed, 01 May 2002 18:41:08 PDT Date: Wed, 1 May 2002 18:41:08 -0700 (PDT) From: David Xu Subject: Re: hlt when idle? To: Terry Lambert , Andrew Gallatin Cc: freebsd-smp@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org >Andrew Gallatin wrote: > Can somebody remind me why we do not hlt in the idle loop on MP x86s? > >Halting with "giant" and/or "the scheduler lock" held is bad. > >> Is this because a HLTed CPU is not going to notice a new runnable job >> (possibly migrating from another CPU) until it gets an interrupt to >> wake it up? > >Partly. It's worse than that, if the system is relatively >quiescent. Interrupts are routed with entry to the kernel, >the system doesn't run in "virtual wire" mode, and it doesn't >support seperately routing interrupts. You can not set "virtual wire" mode when SMP is enabled otherwise every CPU will send interrupt ACK to PIC and make mess the PIC and lost other interrupts. -- David Xu __________________________________________________ Do You Yahoo!? Yahoo! Health - your guide to health and wellness http://health.yahoo.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 21: 7:20 2002 Delivered-To: freebsd-smp@freebsd.org Received: from fw.wemm.org (12-232-135-171.client.attbi.com [12.232.135.171]) by hub.freebsd.org (Postfix) with ESMTP id 7A37B37B416; Wed, 1 May 2002 21:07:17 -0700 (PDT) Received: from overcee.wemm.org (overcee.wemm.org [10.0.0.3]) by fw.wemm.org (8.11.6/8.11.6) with ESMTP id g4247H403899; Wed, 1 May 2002 21:07:17 -0700 (PDT) (envelope-from peter@wemm.org) Received: from wemm.org (localhost [127.0.0.1]) by overcee.wemm.org (Postfix) with ESMTP id CE4BA38CC; Wed, 1 May 2002 21:07:16 -0700 (PDT) (envelope-from peter@wemm.org) X-Mailer: exmh version 2.5 07/13/2001 with nmh-1.0.4 To: Andrew Gallatin Cc: John Baldwin , freebsd-smp@FreeBSD.ORG Subject: Re: hlt when idle? In-Reply-To: <15568.20086.979721.992191@grasshopper.cs.duke.edu> Date: Wed, 01 May 2002 21:07:16 -0700 From: Peter Wemm Message-Id: <20020502040716.CE4BA38CC@overcee.wemm.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Andrew Gallatin wrote: > > John Baldwin writes: > > > > On 01-May-2002 Andrew Gallatin wrote: > > > > > > > > > Can somebody remind me why we do not hlt in the idle loop on MP x86s? > > > Is this because a HLTed CPU is not going to notice a new runnable job > > > (possibly migrating from another CPU) until it gets an interrupt to > > > wake it up? > > > > Yes. > > This seems to be an acceptable "loss" in performance in environments > where cooling is a concern. Is there a deadlock danger? Or is it > just a performance tweak to not HLT SMPs? Would you object to making > it a sysctl (machdep.smp_idle_hlt)? There is no deadlock danger. The only "problem" is that if you have one cpu doing a wakeup() on another process and putting it on the run queue, nothing happens to jolt the other cpu out of the halt state so that it notices that there is work to do. So, on one hand you have the problem where you have a runnable process but the other cpu is asleep till the next interrupt. And on the other hand, right now we have the cpus hammering the cache line that holds the run queue status bits. Since you reported a 15 second speedup in your test, perhaps the spinning cpu is causing more contention than we save by responding immediately to a runnable process being added. IMHO, it is probably worth just putting the halts in and be done with it. Firing off an IPI to wake up cpus is the correct fix though. The catch is that is expensive so you have to know when to do it and when not to. ie: keep some sort of idle/busy state and have the add-run-queue checks notice that they added a new runnable process (vs previously idle) and notice that there is an idle cpu that could pick it up and fire an IPI in its direction. Only experimentation will tell us if this is worth it. Cheers, -Peter -- Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 23:17:58 2002 Delivered-To: freebsd-smp@freebsd.org Received: from falcon.prod.itd.earthlink.net (falcon.mail.pas.earthlink.net [207.217.120.74]) by hub.freebsd.org (Postfix) with ESMTP id 470AB37B417; Wed, 1 May 2002 23:17:50 -0700 (PDT) Received: from pool0052.cvx21-bradley.dialup.earthlink.net ([209.179.192.52] helo=mindspring.com) by falcon.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 1739uX-0007Uc-00; Wed, 01 May 2002 23:17:50 -0700 Message-ID: <3CD0D9F2.839890E7@mindspring.com> Date: Wed, 01 May 2002 23:17:22 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: Andrew Gallatin , freebsd-smp@freebsd.org Subject: Re: hlt when idle? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > > Do both CPUs get clock interrupts on x86? > > No, the interrupts seem to be round-robin, but each clock intr is only > sent to one CPU unlike on alpha where they are broadcast. I was going to make the obvious Alpha comment because it was Andrew asking the question, but since he posted it only to the SMP list and talked about his Pentium III, I didn't. The same is true on SPARC, FWIW. All interrupts on x86 *could* be "broadcast", if the system was run in "virtual wire" mode (MP Spec 1.4). This would *still* break on motherboards that failed to route the clock interrupts via the I/O APIC, though, which is "enough of them to be a large problem". 8-(. Sending an IPI for all processors for the clock interrupt from the processor that receives the clock interrupt is not really a good idea (as I said before), but it's a possible method of achieving "virtual broadcast". As I also said before... "ugly". 8-(. That aside, it might be an OK thing to do on the non-Intel platforms (but again, he was complaining about x86). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 23:25: 0 2002 Delivered-To: freebsd-smp@freebsd.org Received: from falcon.prod.itd.earthlink.net (falcon.mail.pas.earthlink.net [207.217.120.74]) by hub.freebsd.org (Postfix) with ESMTP id 33D9037B41D; Wed, 1 May 2002 23:24:55 -0700 (PDT) Received: from pool0052.cvx21-bradley.dialup.earthlink.net ([209.179.192.52] helo=mindspring.com) by falcon.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 173A1O-0004Kh-00; Wed, 01 May 2002 23:24:54 -0700 Message-ID: <3CD0DB9A.143FCEC7@mindspring.com> Date: Wed, 01 May 2002 23:24:26 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: freebsd-smp@freebsd.org, Andrew Gallatin Subject: Re: hlt when idle? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > On 01-May-2002 Terry Lambert wrote: > > Andrew Gallatin wrote: > >> Can somebody remind me why we do not hlt in the idle loop on MP x86s? > > > > Halting with "giant" and/or "the scheduler lock" held is bad. > > Considering the idle loop doesn't hold any locks this doesn't apply. See the previously referenced discussion. The lock would have to be held, and/or all interrupts would have to be directed to the idle processor. There would still be a wakeup problem for any wakeup that triggered two or more things runnable simultaneously, since only one processor would wake up. There are a couple ways around this, but I think they are too ugly. > >> Do both CPUs get clock interrupts on x86? > > > > The clock interrupts may or may not be routed via the APIC; > > routing of the clock interrupt is one of the most common > > bogosities that you have to handle on x86 SMP systems; there > > is a special printf for complaining about having to work > > around broken BIOS and/or motherboard circuitry. > > You don't know what you are talking about. All the interrupts come > through the APIC. The hack that we do is to abuse the fact that > the normal AT PIC routes through the APIC to piggy back the clock > interrupt onto that (via routing it to the AT PIC). However, the > interrupt still comes to the CPU from the I/O APIc. It was my understanding that the AT PIC had to be directed, and could not run virtual wire. If this isn't correct, then my bad. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed May 1 23:43:42 2002 Delivered-To: freebsd-smp@freebsd.org Received: from swan.prod.itd.earthlink.net (swan.mail.pas.earthlink.net [207.217.120.123]) by hub.freebsd.org (Postfix) with ESMTP id 39FBC37B417; Wed, 1 May 2002 23:43:39 -0700 (PDT) Received: from pool0052.cvx21-bradley.dialup.earthlink.net ([209.179.192.52] helo=mindspring.com) by swan.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 173AJV-0006FK-00; Wed, 01 May 2002 23:43:37 -0700 Message-ID: <3CD0DFFD.20F256C6@mindspring.com> Date: Wed, 01 May 2002 23:43:09 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Andrew Gallatin Cc: John Baldwin , freebsd-smp@FreeBSD.org Subject: Re: hlt when idle? References: <15567.62317.677224.3470@grasshopper.cs.duke.edu> <15568.20086.979721.992191@grasshopper.cs.duke.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Andrew Gallatin wrote: > > > Do both CPUs get clock interrupts on x86? > > > > No, the interrupts seem to be round-robin, but each clock intr is only > > sent to one CPU unlike on alpha where they are broadcast. > > So each CPU gets (1/num_cpu) * hz clock interrupts/sec? Interrupt delivery isn't round-robin. It just *seems* that way. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 0:38:28 2002 Delivered-To: freebsd-smp@freebsd.org Received: from harrier.prod.itd.earthlink.net (harrier.mail.pas.earthlink.net [207.217.120.12]) by hub.freebsd.org (Postfix) with ESMTP id B5BEC37B405 for ; Thu, 2 May 2002 00:38:24 -0700 (PDT) Received: from pool0052.cvx21-bradley.dialup.earthlink.net ([209.179.192.52] helo=mindspring.com) by harrier.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 173BAU-0004NI-00; Thu, 02 May 2002 00:38:22 -0700 Message-ID: <3CD0ECCB.A499BEFC@mindspring.com> Date: Thu, 02 May 2002 00:37:47 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: David Xu Cc: Andrew Gallatin , freebsd-smp@freebsd.org Subject: Re: hlt when idle? References: <20020502014108.24342.qmail@web20908.mail.yahoo.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org David Xu wrote: > >> Is this because a HLTed CPU is not going to notice a new runnable job > >> (possibly migrating from another CPU) until it gets an interrupt to > >> wake it up? > > > >Partly. It's worse than that, if the system is relatively > >quiescent. Interrupts are routed with entry to the kernel, > >the system doesn't run in "virtual wire" mode, and it doesn't > >support seperately routing interrupts. > > You can not set "virtual wire" mode when SMP is enabled otherwise every CPU > will send interrupt ACK to PIC and make mess the PIC and lost other interrupts. UUUUUUUUGGGGGGHHHHHH! Excuse my dyslexia. Crap. Virtual wire mode is for providing a uniprocessor environemtn. I should have said "Symmetric I/O mode", almost everywhere I have ever said "virtual wire mode" in the past. The relevent section for my comment is "3.6.2.3 Symmetric I/O Mode", which says: Some MP operating systems operate in Symmetric I/O Mode. This mode requires at least one I/O APIC to operate. In this mode, I/O interrupts are generated by the I/O APIC. All 8259 interrupt lines are either masked or work together with the I/O APIC in a mixed mode. See Figure 3-5 for an overview of Symmetric I/O Mode. The APIC I/O unit has general-purpose interrupt inputs that can be individually programmed to different operating modes. The I/O APIC interrupt line assignments are system implementation specific. Refer to Chapter 4 for custom implementations and to Chapter 5 for default configurations. The hardware must support a mode of operation in which the system can switch easily to Symmetric I/O mode from PIC or Virtual Wire mode. When the operating system is ready to switch to MP operation, it writes a 01H to the IMCR register, if that register is implemented, and enables I/O APIC Redirection Table entries. The hardware must not require any other action on the part of software to make the transition to Symmetric I/O mode. It should be possible to set DMODE in the APIC ICR (Interrupt Command Register) to 100 and the DSH to 10 (broadcast) for the clock interrupt. This would result in the clock interrupt being sent as an NMI to all processors. This doesn't work if some idiot board designer ran the clock into the ISA PIC, and then ran the ISA PIC into the APIC, instead of running the clock into the APIC directly. THis would mean that you have to take all ISA interrupts or none the same as the clock interrupt, which is a crock. So in answer to Andrew's question: you can't get a broadcast for the clock interrupt on all Intel motherboards, only for some of them. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 0:50:33 2002 Delivered-To: freebsd-smp@freebsd.org Received: from harrier.prod.itd.earthlink.net (harrier.mail.pas.earthlink.net [207.217.120.12]) by hub.freebsd.org (Postfix) with ESMTP id 0CD3137B41B; Thu, 2 May 2002 00:50:04 -0700 (PDT) Received: from pool0052.cvx21-bradley.dialup.earthlink.net ([209.179.192.52] helo=mindspring.com) by harrier.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 173BLZ-0001zO-00; Thu, 02 May 2002 00:49:49 -0700 Message-ID: <3CD0EF79.FC1DA1A8@mindspring.com> Date: Thu, 02 May 2002 00:49:13 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Peter Wemm Cc: Andrew Gallatin , John Baldwin , freebsd-smp@FreeBSD.ORG Subject: Re: hlt when idle? References: <20020502040716.CE4BA38CC@overcee.wemm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Peter Wemm wrote: > IMHO, it is probably worth just putting the halts in and be done with it. > Firing off an IPI to wake up cpus is the correct fix though. The catch is > that is expensive so you have to know when to do it and when not to. ie: > keep some sort of idle/busy state and have the add-run-queue checks notice > that they added a new runnable process (vs previously idle) and notice that > there is an idle cpu that could pick it up and fire an IPI in its > direction. Only experimentation will tell us if this is worth it. Yeah. I looked at doing the IPI thing back in 1996 with Jack Vogel's original FreeBSD SMP code from October of 1995, and it turned into an incredible mess really quickly. I know a lot more now than I did then, but I'd still hesitate to do it, unless I was being paid to bash my head against the wall until the wall broke. 8-(. The one comment I'd make about "just putting the halts in" is that it becomes a progressively worse trade-off the more CPUs there are in the machine. Of course, it's not like anyone is building a lot of large CPU count systems, like back in the Sequent 32 CPU machine days, but I know of at least two 8 CPU boxes that might lose based on adding in the HLT. The real problem is that when you have a crapload of CPUs, you generally have them for a reason, which involves shared state, which means multiple sleepers on shared resources. So you may be missing the wakeup on 6 processes on a shared wait. That can really add up, and it's not going to show up in a heterogenous load, like you get from a "make world". For all we know, a homogeneous load could show negative effects at 4 or even 2 CPUs. 8-(. Maybe it could be made into a compile time option? My preference would be to default it "off", to encourage someone to bash their head on "the right fix" (with a comment to that effect, so that someone doesn't "fix it"; have to have learned something from the ATA write caching default state toggle war...). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 7: 1:40 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail.speakeasy.net (mail14.speakeasy.net [216.254.0.214]) by hub.freebsd.org (Postfix) with ESMTP id A20CC37B423 for ; Thu, 2 May 2002 07:01:22 -0700 (PDT) Received: (qmail 11009 invoked from network); 2 May 2002 14:01:21 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail14.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 2 May 2002 14:01:22 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g42E1KF07540; Thu, 2 May 2002 10:01:21 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <15568.20086.979721.992191@grasshopper.cs.duke.edu> Date: Thu, 02 May 2002 10:00:23 -0400 (EDT) From: John Baldwin To: Andrew Gallatin Subject: RE: hlt when idle? Cc: freebsd-smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 01-May-2002 Andrew Gallatin wrote: > > John Baldwin writes: > > > > On 01-May-2002 Andrew Gallatin wrote: > > > > > > > > > Can somebody remind me why we do not hlt in the idle loop on MP x86s? > > > Is this because a HLTed CPU is not going to notice a new runnable job > > > (possibly migrating from another CPU) until it gets an interrupt to > > > wake it up? > > > > Yes. > > This seems to be an acceptable "loss" in performance in environments > where cooling is a concern. Is there a deadlock danger? Or is it > just a performance tweak to not HLT SMPs? Would you object to making > it a sysctl (machdep.smp_idle_hlt)? Another thing that would be possible would be to try to narrow the race further by keeping track of idle cpu's and sending out IPI's when threads are made runnable. > > > Do both CPUs get clock interrupts on x86? > > > > No, the interrupts seem to be round-robin, but each clock intr is only > > sent to one CPU unlike on alpha where they are broadcast. > > So each CPU gets (1/num_cpu) * hz clock interrupts/sec? Well, I'm not sure how the interrupts are routed exactly, so I would tentatively say "roughly" that many. We ipi everyone else when we get a clock interrupt on x86 to do effective broadcast interrupts. > Drew -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 7:18: 0 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail.speakeasy.net (mail12.speakeasy.net [216.254.0.212]) by hub.freebsd.org (Postfix) with ESMTP id 53C2037B422 for ; Thu, 2 May 2002 07:17:30 -0700 (PDT) Received: (qmail 16843 invoked from network); 2 May 2002 14:17:29 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail12.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 2 May 2002 14:17:29 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g42EHSF07620; Thu, 2 May 2002 10:17:29 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20020501151123.G30080@stylus.haikugeek.com> Date: Thu, 02 May 2002 10:16:31 -0400 (EDT) From: John Baldwin To: Jonathan Mini Subject: Re: hlt when idle? Cc: freebsd-smp@FreeBSD.ORG, Andrew Gallatin Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 01-May-2002 Jonathan Mini wrote: > Andrew Gallatin [gallatin@cs.duke.edu] wrote : >> > No, the interrupts seem to be round-robin, but each clock intr is only >> > sent to one CPU unlike on alpha where they are broadcast. >> >> So each CPU gets (1/num_cpu) * hz clock interrupts/sec? > > Yes, but because the timer is set to num_cpu*hz, each CPU ends up getting > the normal hz interrupts. That's why it runs round-robin but looks like a > broadcast. Eh, are you talking about the Alpha? On x86 we don't do this and have to use IPI's to simulate a broadcast-type deal. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 7:25:42 2002 Delivered-To: freebsd-smp@freebsd.org Received: from 12-234-96-171.client.attbi.com (12-234-96-171.client.attbi.com [12.234.96.171]) by hub.freebsd.org (Postfix) with ESMTP id F187A37B417; Thu, 2 May 2002 07:25:34 -0700 (PDT) Received: by 12-234-96-171.client.attbi.com (Postfix, from userid 1000) id 1817BA900; Thu, 2 May 2002 07:29:49 -0700 (PDT) Date: Thu, 2 May 2002 07:29:49 -0700 From: Jonathan Mini To: John Baldwin Cc: freebsd-smp@FreeBSD.org, Andrew Gallatin Subject: Re: hlt when idle? Message-ID: <20020502072949.C56560@stylus.haikugeek.com> References: <20020501151123.G30080@stylus.haikugeek.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: ; from jhb@FreeBSD.org on Thu, May 02, 2002 at 10:16:31AM -0400 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin [jhb@FreeBSD.org] wrote : > > On 01-May-2002 Jonathan Mini wrote: > > Andrew Gallatin [gallatin@cs.duke.edu] wrote : > >> > No, the interrupts seem to be round-robin, but each clock intr is only > >> > sent to one CPU unlike on alpha where they are broadcast. > >> > >> So each CPU gets (1/num_cpu) * hz clock interrupts/sec? > > > > Yes, but because the timer is set to num_cpu*hz, each CPU ends up getting > > the normal hz interrupts. That's why it runs round-robin but looks like a > > broadcast. > > Eh, are you talking about the Alpha? On x86 we don't do this and have to use > IPI's to simulate a broadcast-type deal. > I am obviously thinking about some other SMP implementation, but I have no idea which one. Somebody, somewhere, sets the routing of the clock interrupt to be delivered in a round-robin fashion, and then multiplies the clock frequency by the number of processors. They're really proud of this solution, because (they claim) it reduces contentions of clock-triggered events across processors. Maybe it was Sun? -- Jonathan Mini http://www.haikugeek.com "He who is not aware of his ignorance will be only misled by his knowledge." -- Richard Whatley To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 7:47:11 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail.speakeasy.net (mail14.speakeasy.net [216.254.0.214]) by hub.freebsd.org (Postfix) with ESMTP id 6E90B37B428 for ; Thu, 2 May 2002 07:46:12 -0700 (PDT) Received: (qmail 8335 invoked from network); 2 May 2002 14:46:08 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail14.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 2 May 2002 14:46:08 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g42Ek0F07738; Thu, 2 May 2002 10:46:00 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <20020502072949.C56560@stylus.haikugeek.com> Date: Thu, 02 May 2002 10:45:01 -0400 (EDT) From: John Baldwin To: Jonathan Mini Subject: Re: hlt when idle? Cc: Andrew Gallatin , freebsd-smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 02-May-2002 Jonathan Mini wrote: > John Baldwin [jhb@FreeBSD.org] wrote : > >> >> On 01-May-2002 Jonathan Mini wrote: >> > Andrew Gallatin [gallatin@cs.duke.edu] wrote : >> >> > No, the interrupts seem to be round-robin, but each clock intr is only >> >> > sent to one CPU unlike on alpha where they are broadcast. >> >> >> >> So each CPU gets (1/num_cpu) * hz clock interrupts/sec? >> > >> > Yes, but because the timer is set to num_cpu*hz, each CPU ends up getting >> > the normal hz interrupts. That's why it runs round-robin but looks like a >> > broadcast. >> >> Eh, are you talking about the Alpha? On x86 we don't do this and have to >> use >> IPI's to simulate a broadcast-type deal. >> > > I am obviously thinking about some other SMP implementation, but I have no > idea which one. Somebody, somewhere, sets the routing of the clock interrupt > to be delivered in a round-robin fashion, and then multiplies the clock > frequency by the number of processors. They're really proud of this solution, > because (they claim) it reduces contentions of clock-triggered events across > processors. It probably does. > Maybe it was Sun? Maybe? On the alpha it is very nice, because not only is it broadcast, but it's broadcast in a staggered fashion, so not all CPU's get the clock interrupt at the same time, thus reducing contention. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 9:56:55 2002 Delivered-To: freebsd-smp@freebsd.org Received: from avocet.prod.itd.earthlink.net (avocet.mail.pas.earthlink.net [207.217.120.50]) by hub.freebsd.org (Postfix) with ESMTP id 1360037B416; Thu, 2 May 2002 09:56:51 -0700 (PDT) Received: from pool0542.cvx22-bradley.dialup.earthlink.net ([209.179.200.32] helo=mindspring.com) by avocet.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 173Jsv-0004EU-00; Thu, 02 May 2002 09:56:49 -0700 Message-ID: <3CD16FB4.17BC09B@mindspring.com> Date: Thu, 02 May 2002 09:56:20 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: Andrew Gallatin , freebsd-smp@FreeBSD.org Subject: Re: hlt when idle? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > > This seems to be an acceptable "loss" in performance in environments > > where cooling is a concern. Is there a deadlock danger? Or is it > > just a performance tweak to not HLT SMPs? Would you object to making > > it a sysctl (machdep.smp_idle_hlt)? > > Another thing that would be possible would be to try to narrow the race > further by keeping track of idle cpu's and sending out IPI's when threads > are made runnable. I suggested this already, but recommended against it. So did Peter. Like I said before, this has the problem of spurious IPI delivery to already running processors (if done right; if done wrong, you can still miss IPIs). The problem is that you can't close a race window on knowing the HLT state and being in the HLT state, because there's not a cmpxchnghlt instruction. You can only cause an overlap, so you get spurious notification instead of no notification. > > So each CPU gets (1/num_cpu) * hz clock interrupts/sec? > > Well, I'm not sure how the interrupts are routed exactly, so I would > tentatively say "roughly" that many. We ipi everyone else when we get > a clock interrupt on x86 to do effective broadcast interrupts. The clock interrupt gets sent to the ISA PIC, and maybe also to the APIC. The ISA PIC routes its interrupt to a single APIC input. THis is why if you mask ISA interrupts at the APIC, you can *lose* them. If the clock comes into an APIC input directly, instead of coming in indirectly via the ISA PIC, then it's routing is controlled by the APIC ICR entry for the APIC input. You should really read the Intel MP Specification, v1.4 (it was long enough ago that I did that it was easy for me to screw up my use of their terminology and mix up "virtual wire mode" with "symmetric I/O mode" because of the definition of the words in the title of the mode vs. Intel's definition; I can imagin others would have a similar problem unless they read the spec.). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 9:58:39 2002 Delivered-To: freebsd-smp@freebsd.org Received: from avocet.prod.itd.earthlink.net (avocet.mail.pas.earthlink.net [207.217.120.50]) by hub.freebsd.org (Postfix) with ESMTP id C29E037B41C; Thu, 2 May 2002 09:58:35 -0700 (PDT) Received: from pool0542.cvx22-bradley.dialup.earthlink.net ([209.179.200.32] helo=mindspring.com) by avocet.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 173Jud-0006hT-00; Thu, 02 May 2002 09:58:35 -0700 Message-ID: <3CD1701E.7F0F4CB4@mindspring.com> Date: Thu, 02 May 2002 09:58:06 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: Jonathan Mini , freebsd-smp@FreeBSD.ORG, Andrew Gallatin Subject: Re: hlt when idle? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > >> So each CPU gets (1/num_cpu) * hz clock interrupts/sec? > > > > Yes, but because the timer is set to num_cpu*hz, each CPU ends up getting > > the normal hz interrupts. That's why it runs round-robin but looks like a > > broadcast. > > Eh, are you talking about the Alpha? On x86 we don't do this and have to use > IPI's to simulate a broadcast-type deal. You can get real broadcast behaviour *IF* the clock interrupt comes directly into the APIC, instead of being cascaded through the ISA PIC. That's kind of the whole point: some motherboards do, some motherboards don't. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 10: 1:39 2002 Delivered-To: freebsd-smp@freebsd.org Received: from avocet.prod.itd.earthlink.net (avocet.mail.pas.earthlink.net [207.217.120.50]) by hub.freebsd.org (Postfix) with ESMTP id 024FD37B41B; Thu, 2 May 2002 10:01:37 -0700 (PDT) Received: from pool0542.cvx22-bradley.dialup.earthlink.net ([209.179.200.32] helo=mindspring.com) by avocet.prod.itd.earthlink.net with esmtp (Exim 3.33 #2) id 173JxZ-0003Gs-00; Thu, 02 May 2002 10:01:37 -0700 Message-ID: <3CD170D4.48AEB353@mindspring.com> Date: Thu, 02 May 2002 10:01:08 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Jonathan Mini Cc: John Baldwin , freebsd-smp@FreeBSD.org, Andrew Gallatin Subject: Re: hlt when idle? References: <20020501151123.G30080@stylus.haikugeek.com> <20020502072949.C56560@stylus.haikugeek.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Jonathan Mini wrote: > I am obviously thinking about some other SMP implementation, but I have no > idea which one. Somebody, somewhere, sets the routing of the clock interrupt > to be delivered in a round-robin fashion, and then multiplies the clock > frequency by the number of processors. They're really proud of this solution, > because (they claim) it reduces contentions of clock-triggered events across > processors. > > Maybe it was Sun? You are an ex-Be-geek. Maybe it was the BeBox and the two processor PPC603e box from Apple? From memory, in addition to using MEI instead of MESI cache coherency because the CPUs were not built for SMP (the MEI arbitration was done by putting contention hardware in place of the L2 cache), interrupt routing was hardware round-robin. 8-). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 10: 4:56 2002 Delivered-To: freebsd-smp@freebsd.org Received: from hawk.mail.pas.earthlink.net (hawk.mail.pas.earthlink.net [207.217.120.22]) by hub.freebsd.org (Postfix) with ESMTP id E0EC237B42B; Thu, 2 May 2002 10:04:05 -0700 (PDT) Received: from pool0542.cvx22-bradley.dialup.earthlink.net ([209.179.200.32] helo=mindspring.com) by hawk.mail.pas.earthlink.net with esmtp (Exim 3.33 #2) id 173Jzx-0000vx-00; Thu, 02 May 2002 10:04:05 -0700 Message-ID: <3CD17167.96118A88@mindspring.com> Date: Thu, 02 May 2002 10:03:35 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: Jonathan Mini , Andrew Gallatin , freebsd-smp@FreeBSD.org Subject: Re: hlt when idle? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > On 02-May-2002 Jonathan Mini wrote: > > They're really proud of this solution, > > because (they claim) it reduces contentions of clock-triggered events across > > processors. > > It probably does. It definitely does, provided you are guaranteed that the clock interrupt handler can always execute in less time than the interval between clock interrupts. Then you don't have to contend on the global handler lock to ensure it only runs once. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 10:20: 3 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail.speakeasy.net (mail12.speakeasy.net [216.254.0.212]) by hub.freebsd.org (Postfix) with ESMTP id 623BE37B41D for ; Thu, 2 May 2002 10:19:41 -0700 (PDT) Received: (qmail 7850 invoked from network); 2 May 2002 17:19:40 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail12.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 2 May 2002 17:19:40 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g42HJeF08274; Thu, 2 May 2002 13:19:40 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3CD17167.96118A88@mindspring.com> Date: Thu, 02 May 2002 13:18:42 -0400 (EDT) From: John Baldwin To: Terry Lambert Subject: Re: hlt when idle? Cc: freebsd-smp@FreeBSD.org, Andrew Gallatin , Jonathan Mini Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 02-May-2002 Terry Lambert wrote: > John Baldwin wrote: >> On 02-May-2002 Jonathan Mini wrote: >> > They're really proud of this solution, >> > because (they claim) it reduces contentions of clock-triggered events >> > across >> > processors. >> >> It probably does. > > It definitely does, provided you are guaranteed that the clock > interrupt handler can always execute in less time than the > interval between clock interrupts. Then you don't have to > contend on the global handler lock to ensure it only runs once. Well, with our current interrupts the foo_process() (should be foo_thread()) versions still need the global sched_lock to modify some flag fields and what not. (Some of this could be eliminated by splitting td_flags into two parts, one that needs the lock and one that doesn't.) Thus, if they all fire at once, even if it's broadcast they all contend with each other, wherease if they are broadcast but staggered such as on the alpha they don't in theory contend with each other at all. (Provided the handler is fast enough, which it should be.) > -- Terry -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 10:32:47 2002 Delivered-To: freebsd-smp@freebsd.org Received: from hawk.mail.pas.earthlink.net (hawk.mail.pas.earthlink.net [207.217.120.22]) by hub.freebsd.org (Postfix) with ESMTP id 5DD0737B41A; Thu, 2 May 2002 10:32:25 -0700 (PDT) Received: from pool0542.cvx22-bradley.dialup.earthlink.net ([209.179.200.32] helo=mindspring.com) by hawk.mail.pas.earthlink.net with esmtp (Exim 3.33 #2) id 173KRM-00028R-00; Thu, 02 May 2002 10:32:24 -0700 Message-ID: <3CD1780B.2CC97DDB@mindspring.com> Date: Thu, 02 May 2002 10:31:55 -0700 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: freebsd-smp@FreeBSD.org, Andrew Gallatin , Jonathan Mini Subject: Re: hlt when idle? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > > It definitely does, provided you are guaranteed that the clock > > interrupt handler can always execute in less time than the > > interval between clock interrupts. Then you don't have to > > contend on the global handler lock to ensure it only runs once. > > Well, with our current interrupts the foo_process() (should be foo_thread()) > versions still need the global sched_lock to modify some flag fields and what > not. (Some of this could be eliminated by splitting td_flags into two parts, > one that needs the lock and one that doesn't.) Thus, if they all fire at once, > even if it's broadcast they all contend with each other, wherease if they are > broadcast but staggered such as on the alpha they don't in theory contend with > each other at all. (Provided the handler is fast enough, which it should be.) Ah. I omitted a piece because it was obvious to me, when in retrospect it's really non-obvious. You reprogram the top 32 bits of the ICR in the APIC for the interrupt each time as part of the interrupt handler, so it's actually unicast to a single processor to get the "round robin" effect ("bother that CPU instead of me next time"). IMO, going round-robin on this sort of thing is a bad idea; L1 and L2 cache busting are a biggie for me -- but then I want to scale to an infinite number of processors. 8-). If you wanted to broadcast a clock interrupt, which makes more sense for me, then the system "clock interrupt" processing would be handled on a single CPU. Alternately, you could rewrite the upper 32 bits, rewrite the lower 32 bits (which would trigger the IPI to be sent), and then rewrite the upper 32 bits again, to target the next guy in line. I think the confusion is because it's useful for the clock to be broadcast for one set of reasons, and it's useful to hit on only one processor for global handler processing for another set of reasons. I think a lot of the desire to hit a CPU over the head to have it look at the scheduler queue comes because the scheduler queues aren't per-CPU, and thus CPU migration is not a really rare event, like it should be (in order to get CPU affinity). In the migration case, an IPI targetting only the migrated-to CPU, only if it's HLT'ed, is probably acceptable overhead. Actually, here's an incredibly good reference for Intel APIC and SMP processing, from Purdue University: Multiprocessing Support for Hobby OSes Explained Ben L. Titzer http://expert.cc.purdue.edu/~titzer/redpants/mpx86.html It goes into the details of ICR programming, which is where the black magic is for broadcast vs. non-broadcast, etc.. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 11: 1:35 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail.speakeasy.net (mail11.speakeasy.net [216.254.0.211]) by hub.freebsd.org (Postfix) with ESMTP id ADADC37B41A for ; Thu, 2 May 2002 11:01:31 -0700 (PDT) Received: (qmail 22217 invoked from network); 2 May 2002 18:01:30 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail11.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 2 May 2002 18:01:30 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g42I1QF08432; Thu, 2 May 2002 14:01:26 -0400 (EDT) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3CD1780B.2CC97DDB@mindspring.com> Date: Thu, 02 May 2002 14:00:27 -0400 (EDT) From: John Baldwin To: Terry Lambert Subject: Re: hlt when idle? Cc: Jonathan Mini , Andrew Gallatin , freebsd-smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 02-May-2002 Terry Lambert wrote: > I think the confusion is because it's useful for the clock to > be broadcast for one set of reasons, and it's useful to hit on > only one processor for global handler processing for another > set of reasons. Err, some confusion may be that in current we have two different types of clock interrupt handlers. :) For example, for hardclock, we ahve hardclock() and hardclock_process(). We call hardclock() on only one CPU, and hardclock_process() on all the others. hardclock() does system-wide global changes and then runs the thread-specific hardclock_process(). hardclock_process() does thread-specific things like setting up SIGVTALRM and SIGPROF. Currently hardclock_process() (and even more-so, statclock_process()) use the global sched_lock, thus broadcast doesn't reduce contention unless it is a staggered broadcast. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu May 2 19: 4:29 2002 Delivered-To: freebsd-smp@freebsd.org Received: from 12-234-96-171.client.attbi.com (12-234-96-171.client.attbi.com [12.234.96.171]) by hub.freebsd.org (Postfix) with ESMTP id 88A3737B400; Thu, 2 May 2002 19:04:23 -0700 (PDT) Received: by 12-234-96-171.client.attbi.com (Postfix, from userid 1000) id ADC75A900; Thu, 2 May 2002 19:08:28 -0700 (PDT) Date: Thu, 2 May 2002 19:08:28 -0700 From: Jonathan Mini To: Terry Lambert Cc: John Baldwin , freebsd-smp@FreeBSD.org, Andrew Gallatin Subject: Re: hlt when idle? Message-ID: <20020502190828.D56560@stylus.haikugeek.com> References: <20020501151123.G30080@stylus.haikugeek.com> <20020502072949.C56560@stylus.haikugeek.com> <3CD170D4.48AEB353@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <3CD170D4.48AEB353@mindspring.com>; from tlambert2@mindspring.com on Thu, May 02, 2002 at 10:01:08AM -0700 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Terry Lambert [tlambert2@mindspring.com] wrote : > Jonathan Mini wrote: > > Maybe it was Sun? > > You are an ex-Be-geek. Maybe it was the BeBox and the two > processor PPC603e box from Apple? That crossed my mind first, but Be wired all of the interrupts to CPU 0 all the time, including the clock interrupt. It uses IPIs to let the other processors know of events like VM modifications and scheduling excess. > >From memory, in addition to using MEI instead of MESI cache > coherency because the CPUs were not built for SMP (the MEI > arbitration was done by putting contention hardware in place > of the L2 cache), interrupt routing was hardware round-robin. > > 8-). That is more detail than I know about the PPC-based BeBox. By the time I joined Be, we had already officially dropped support for them and were an x86 shop. -- Jonathan Mini http://www.haikugeek.com "He who is not aware of his ignorance will be only misled by his knowledge." -- Richard Whatley To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message