From owner-freebsd-smp Sun Aug 8 10:58:10 1999 Delivered-To: freebsd-smp@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id 03D9614F81; Sun, 8 Aug 1999 10:58:06 -0700 (PDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (harmony.village.org [10.0.0.6]) by rover.village.org (8.9.3/8.9.3) with ESMTP id LAA10214; Sun, 8 Aug 1999 11:56:30 -0600 (MDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id LAA01076; Sun, 8 Aug 1999 11:56:19 -0600 (MDT) Message-Id: <199908081756.LAA01076@harmony.village.org> To: tcobb@staff.circle.net Subject: Re: continued crashes with 3.1-Stable Cc: jkh@zippy.cdrom.com, freebsd-smp@FreeBSD.ORG, lightningweb@hotmail.com, freebsd-stable@FreeBSD.ORG In-reply-to: Your message of "Sat, 07 Aug 1999 08:28:10 EDT." <307D63ED6749CF11AAE9005004461A5B3FB9@FREYA> References: <307D63ED6749CF11AAE9005004461A5B3FB9@FREYA> Date: Sun, 08 Aug 1999 11:56:19 -0600 From: Warner Losh Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In message <307D63ED6749CF11AAE9005004461A5B3FB9@FREYA> tcobb@staff.circle.net writes: : Unfortunately, our hosting customer who is running a high-end : e-commerce site on this machine isn't excited about being : a testbed, they want a solution that works and stays working. Are you using loopback NFS mounts? If so, there is a bug under heavy load with that... Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Aug 8 11:13:26 1999 Delivered-To: freebsd-smp@freebsd.org Received: from ns.mt.sri.com (ns.mt.sri.com [206.127.79.91]) by hub.freebsd.org (Postfix) with ESMTP id 5DE5A14E3C for ; Sun, 8 Aug 1999 11:13:13 -0700 (PDT) (envelope-from nate@mt.sri.com) Received: from mt.sri.com (rocky.mt.sri.com [206.127.76.100]) by ns.mt.sri.com (8.8.8/8.8.8) with SMTP id MAA00298; Sun, 8 Aug 1999 12:09:11 -0600 (MDT) (envelope-from nate@rocky.mt.sri.com) Received: by mt.sri.com (SMI-8.6/SMI-SVR4) id MAA16849; Sun, 8 Aug 1999 12:09:04 -0600 Date: Sun, 8 Aug 1999 12:09:04 -0600 Message-Id: <199908081809.MAA16849@mt.sri.com> From: Nate Williams MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit To: Kazutaka YOKOTA Cc: Nate Williams , freebsd-smp@freebsd.org Subject: Re: SMP + XDM = keyboard lockup In-Reply-To: <199908070629.PAA10251@zodiac.mech.utsunomiya-u.ac.jp> References: <199907221817.LAA09698@usr05.primenet.com> <199907230402.NAA02136@zodiac.mech.utsunomiya-u.ac.jp> <199907231624.KAA07741@mt.sri.com> <199908070629.PAA10251@zodiac.mech.utsunomiya-u.ac.jp> X-Mailer: VM 6.34 under 19.16 "Lille" XEmacs Lucid Reply-To: nate@mt.sri.com (Nate Williams) Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > >> I guess it may not. After init forks and execs getty and xdm, we > >> don't know the exact order in which these processes will be scheduled > >> to run on which CPU, do we? /etc/ttys does not ensure the order of > >> execution. > >> > >> If xdm happens to run before other getty processes, it may grab a vty > >> which will be eventually opened by a getty later. > > > >Actually, I had forgotten about this. This is true because the line in > >/etc/ttys doesn't specify *which* tty for X to use. If we could easily > >force X to use that specific tty, it would avoid this problem. > > > >Nate > > We can specify the vty number in /usr/X11R6/lib/X11/xdm/Xservers as: > > :0 local /usr/X11R6/bin/X vt4 > > can we not? Yes, but that kind of defeats the purpose of having a 'single' entry in /etc/ttys that allows us to set the correct tty. Editing two files means that more often than not it won't be done correctly. > I know this is troublesome in the sense that if we want to change the > vty for X, we need to update both /etc/tty and > /usr/X11R6/lib/X11/xdm/Xservers ;-< Agreed. :( > In any case, the race condition regarding xdm and getty which we > discussed in this mailing list should be reflected in FAQ. Our FAQ > currently has an entry for xdm; it gives you the impression that the > race condition exists only when we start xdm from one of rc.* files, > and people may think there should be no problem when xdm is started > from /etc/ttys. Agreed. > ------------------------------------------------------------------------------ > 9.7 How do I start XDM on boot? > > There are two schools of thought on how to start xdm. One school starts xdm > from /etc/ttys using the supplied example, while the other simply runs xdm from > rc.local or from a X.sh script in /usr/local/etc/rc.d. Both are equally valid, > and one may work in situations where the other doesn't. In both cases the > result is the same: X will popup a graphical login: prompt. > > The ttys method has the advantage of documenting which vty X will start on and > passing the responsibility of restarting the X server on logout to init. The > rc.local method makes it easy to kill xdm if there is a problem starting the X > server. > > If loaded from rc.local, xdm should be started without any arguments (i.e., as > a daemon). xdm must start AFTER getty runs, or else gettty and xdm will con- > flict, locking out the console. The best way around this is to have the script > sleep 10 seconds or so then launch xdm. > > A previous version of the FAQ said to add the vt you want X to use to the > /usr/X11R6/lib/X11/xdm/Xservers file. This is not necessary: X will use the > first free vt it finds. > ------------------------------------------------------------------------------ > > The last paragraph should be modified to something like: > > "If you are to start xdm from /etc/ttys, there still is a chance of > conflict between xdm and getty. One way to avoid this is to add the > vt number in the /usr/X11R6/lib/X11/xdm/Xservers file. > > :0 local /usr/X11R6/bin/X vt4 > > The above example will direct the X server to run in /dev/ttyv3. > Note the number is offset by one. The X server counts the vt from one, > whereas the FreeBSD kernel numbers vt from zero." > > Comments? Sounds good to me. Commit away! Nate To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Aug 8 12: 1:36 1999 Delivered-To: freebsd-smp@freebsd.org Received: from freya.circle.net (morrigu.circle.net [209.95.64.11]) by hub.freebsd.org (Postfix) with ESMTP id ECD2C14BD6; Sun, 8 Aug 1999 12:01:32 -0700 (PDT) (envelope-from tcobb@staff.circle.net) Received: by FREYA with Internet Mail Service (5.5.2448.0) id ; Sun, 8 Aug 1999 14:55:27 -0400 Message-ID: <307D63ED6749CF11AAE9005004461A5B3FDF@FREYA> From: tcobb@staff.circle.net To: imp@village.org Cc: jkh@zippy.cdrom.com, freebsd-smp@FreeBSD.ORG, lightningweb@hotmail.com, freebsd-stable@FreeBSD.ORG Subject: RE: continued crashes with 3.1-Stable Date: Sun, 8 Aug 1999 14:55:26 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2448.0) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I'm not using loopback nfs mounts (though I am using those successfully on other UP machines). I think I've found something that may explain what I'm seeing (the freezes): http://www.hardwarecentral.com/hardwarecentral/reports/881/1 -Troy Cobb Circle Net, Inc. http://www.circle.net > -----Original Message----- > From: Warner Losh [mailto:imp@village.org] > Sent: Sunday, August 08, 1999 1:56 PM > To: tcobb@staff.circle.net > Cc: jkh@zippy.cdrom.com; freebsd-smp@FreeBSD.ORG; > lightningweb@hotmail.com; freebsd-stable@FreeBSD.ORG > Subject: Re: continued crashes with 3.1-Stable > > > In message <307D63ED6749CF11AAE9005004461A5B3FB9@FREYA> > tcobb@staff.circle.net writes: > : Unfortunately, our hosting customer who is running a high-end > : e-commerce site on this machine isn't excited about being > : a testbed, they want a solution that works and stays working. > > Are you using loopback NFS mounts? If so, there is a bug > under heavy > load with that... > > Warner > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Aug 8 12: 5:41 1999 Delivered-To: freebsd-smp@freebsd.org Received: from rover.village.org (rover.village.org [204.144.255.49]) by hub.freebsd.org (Postfix) with ESMTP id DA92A14BD6; Sun, 8 Aug 1999 12:05:28 -0700 (PDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (harmony.village.org [10.0.0.6]) by rover.village.org (8.9.3/8.9.3) with ESMTP id NAA10349; Sun, 8 Aug 1999 13:02:14 -0600 (MDT) (envelope-from imp@harmony.village.org) Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id NAA01304; Sun, 8 Aug 1999 13:02:04 -0600 (MDT) Message-Id: <199908081902.NAA01304@harmony.village.org> To: tcobb@staff.circle.net Subject: Re: continued crashes with 3.1-Stable Cc: jkh@zippy.cdrom.com, freebsd-smp@FreeBSD.ORG, lightningweb@hotmail.com, freebsd-stable@FreeBSD.ORG In-reply-to: Your message of "Sun, 08 Aug 1999 14:55:26 EDT." <307D63ED6749CF11AAE9005004461A5B3FDF@FREYA> References: <307D63ED6749CF11AAE9005004461A5B3FDF@FREYA> Date: Sun, 08 Aug 1999 13:02:03 -0600 From: Warner Losh Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In message <307D63ED6749CF11AAE9005004461A5B3FDF@FREYA> tcobb@staff.circle.net writes: : I'm not using loopback nfs mounts (though I am using those : successfully on other UP machines). That's an accident waiting to happen. If you are doing anything with a high i/o load, you'll hit a deadlock in the implementation of nfs on -stable only. try creating a 16M-32M file and copying it around a few times. Kills my -stable machines every time... Warner To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Aug 11 8: 1:29 1999 Delivered-To: freebsd-smp@freebsd.org Received: from hercules.cs.ucsb.edu (hercules.cs.ucsb.edu [128.111.41.30]) by hub.freebsd.org (Postfix) with ESMTP id 7F14A15330 for ; Wed, 11 Aug 1999 08:01:23 -0700 (PDT) (envelope-from blanquer@cs.ucsb.edu) Received: from ella (ella [128.111.49.42]) by hercules.cs.ucsb.edu (8.8.6/8.8.6) with ESMTP id IAA28205 for ; Wed, 11 Aug 1999 08:01:15 -0700 (PDT) Date: Wed, 11 Aug 1999 08:01:14 -0700 (PDT) From: "Josep Maria M. Blanquer" X-Sender: blanquer@ella To: freebsd-smp@freebsd.org Subject: Questions Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, I've been taking a look on the SMP implementation on FreeBSD and there's something that I'm probably missing... I've read some docs about thoughts and implementation ideas: (http://www.freebsd.org/~fsmp/SMP/SMP.html ) but I'm not aware of any other documentation, is there any more complete/up-to-date one? The version I've been looking at is FreeBSD-stable (of 2-3 weeks ago). The things I don't fully understant/I'm missing are: 1- On the IO APIC configuration. When the IO APIC/(s) are configured to forward the interrupts to the local APICs, the delivery mode is set to the lowest priority and the destination mode is set up to physical! Is that the way it should go? From my understanding if you specify physical dest, you can only specify a single APIC ID right? So that means that all INT are always forwarded to the BSP (id 0)? 2- The other question is the use of lastcpu field on proc structure: I've seen that the field is being updated on cpu_switch() but I couldn't find any place where the field is being used. Is this a pending work? I mean there is or has been any work on SMP specific scheduling? I mean, process balancing, local/global queues and so on? Thanks, Josep M. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Aug 11 10: 7:42 1999 Delivered-To: freebsd-smp@freebsd.org Received: from sasami.jurai.net (sasami.jurai.net [63.67.141.99]) by hub.freebsd.org (Postfix) with ESMTP id 959261557C for ; Wed, 11 Aug 1999 10:07:39 -0700 (PDT) (envelope-from winter@jurai.net) Received: from localhost (winter@localhost) by sasami.jurai.net (8.8.8/8.8.7) with ESMTP id NAA02989; Wed, 11 Aug 1999 13:07:08 -0400 (EDT) Date: Wed, 11 Aug 1999 13:07:07 -0400 (EDT) From: "Matthew N. Dodd" To: freebsd-smp@freebsd.org, matts-hardware@jurai.net Subject: New list: ALR 6x6/Unisys HS/6 hardware. Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I've created a new list for talking about the above boards. Email majordomo@halo.net -> subscribe sixway-tech Several people including myself already have these boards and others are witing for theirs to arrive. I figure we can start hashing out some of the shared issues... -- | Matthew N. Dodd | '78 Datsun 280Z | '75 Volvo 164E | FreeBSD/NetBSD | | winter@jurai.net | 2 x '84 Volvo 245DL | ix86,sparc,pmax | | http://www.jurai.net/~winter | This Space For Rent | ISO8802.5 4ever | To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Aug 11 22:40:14 1999 Delivered-To: freebsd-smp@freebsd.org Received: from mail.rdc1.sfba.home.com (ha1.rdc1.sfba.home.com [24.0.0.66]) by hub.freebsd.org (Postfix) with ESMTP id 5CB7714D81 for ; Wed, 11 Aug 1999 22:40:05 -0700 (PDT) (envelope-from adsharma@c62443-a.frmt1.sfba.home.com) Received: from c62443-a.frmt1.sfba.home.com ([24.0.69.165]) by mail.rdc1.sfba.home.com (InterMail v4.01.01.00 201-229-111) with ESMTP id <19990812053855.URA8807.mail.rdc1.sfba.home.com@c62443-a.frmt1.sfba.home.com>; Wed, 11 Aug 1999 22:38:55 -0700 Received: (from adsharma@localhost) by c62443-a.frmt1.sfba.home.com (8.9.3/8.9.3) id AAA14491; Thu, 12 Aug 1999 00:38:48 -0700 Date: Thu, 12 Aug 1999 00:38:48 -0700 From: Arun Sharma To: "Josep Maria M. Blanquer" Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Questions Message-ID: <19990812003848.A14472@home.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.95.6i In-Reply-To: ; from Josep Maria M. Blanquer on Wed, Aug 11, 1999 at 08:01:14AM -0700 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Wed, Aug 11, 1999 at 08:01:14AM -0700, Josep Maria M. Blanquer wrote: > The things I don't fully understant/I'm missing are: > 1- On the IO APIC configuration. When the IO APIC/(s) are configured > to forward the interrupts to the local APICs, the delivery mode > is set to the lowest priority and the destination mode is set up > to physical! Is that the way it should go? From my understanding > if you specify physical dest, you can only specify a single APIC > ID right? So that means that all INT are always forwarded to the > BSP (id 0)? Wrong. A physical destination ID of all 1s means broadcast to all local APICs. Refer to Chapter 7.4.9.1 (Pentium III manual, Vol 3, systems programming guide) -Arun To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Aug 12 15: 5:39 1999 Delivered-To: freebsd-smp@freebsd.org Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (Postfix) with ESMTP id 4B3FF14D17 for ; Thu, 12 Aug 1999 15:05:35 -0700 (PDT) (envelope-from tlambert@usr04.primenet.com) Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id PAA15691; Thu, 12 Aug 1999 15:05:34 -0700 (MST) Received: from usr04.primenet.com(206.165.6.204) via SMTP by smtp04.primenet.com, id smtpdAAAESaiNE; Thu Aug 12 15:05:29 1999 Received: (from tlambert@localhost) by usr04.primenet.com (8.8.5/8.8.5) id PAA20060; Thu, 12 Aug 1999 15:05:34 -0700 (MST) From: Terry Lambert Message-Id: <199908122205.PAA20060@usr04.primenet.com> Subject: Re: Questions To: adsharma@home.com (Arun Sharma) Date: Thu, 12 Aug 1999 22:05:34 +0000 (GMT) Cc: blanquer@cs.ucsb.edu, freebsd-smp@FreeBSD.ORG In-Reply-To: <19990812003848.A14472@home.com> from "Arun Sharma" at Aug 12, 99 00:38:48 am X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > On Wed, Aug 11, 1999 at 08:01:14AM -0700, Josep Maria M. Blanquer wrote: > > > The things I don't fully understant/I'm missing are: > > 1- On the IO APIC configuration. When the IO APIC/(s) are configured > > to forward the interrupts to the local APICs, the delivery mode > > is set to the lowest priority and the destination mode is set up > > to physical! Is that the way it should go? From my understanding > > if you specify physical dest, you can only specify a single APIC > > ID right? So that means that all INT are always forwarded to the > > BSP (id 0)? > > Wrong. A physical destination ID of all 1s means broadcast to all local > APICs. > > Refer to Chapter 7.4.9.1 (Pentium III manual, Vol 3, systems programming guide) See also the current version of the Intel Multiprocessing Specification, particularly the section on "Virtual Wire Mode" for interrupt delivery, and why an OS that doesn't specify "Virtual Wire Mode" of operation is not actually symmetric. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Aug 12 16:17: 8 1999 Delivered-To: freebsd-smp@freebsd.org Received: from cs.rice.edu (cs.rice.edu [128.42.1.30]) by hub.freebsd.org (Postfix) with ESMTP id 9CD1214CC9 for ; Thu, 12 Aug 1999 16:16:57 -0700 (PDT) (envelope-from alc@cs.rice.edu) Received: from nonpc.cs.rice.edu (nonpc.cs.rice.edu [128.42.1.219]) by cs.rice.edu (8.9.0/8.9.0) with ESMTP id SAA06864; Thu, 12 Aug 1999 18:17:00 -0500 (CDT) Received: (from alc@localhost) by nonpc.cs.rice.edu (8.9.3/8.7.3) id SAA27389; Thu, 12 Aug 1999 18:17:00 -0500 (CDT) Date: Thu, 12 Aug 1999 18:17:00 -0500 From: Alan Cox To: "Josep Maria M. Blanquer" Cc: freebsd-smp@freebsd.org Subject: Re: Questions Message-ID: <19990812181700.B27359@nonpc.cs.rice.edu> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.95.1i In-Reply-To: ; from Josep Maria M. Blanquer on Wed, Aug 11, 1999 at 08:01:14AM -0700 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Wed, Aug 11, 1999 at 08:01:14AM -0700, Josep Maria M. Blanquer wrote: > ... > > 2- The other question is the use of lastcpu field on proc structure: > I've seen that the field is being updated on cpu_switch() but > I couldn't find any place where the field is being used. Is this > a pending work? I mean there is or has been any work on SMP > specific scheduling? I mean, process balancing, local/global queues > and so on? > Try asking John Dyson (dyson@iquest.net). I think he has experimented with some limited forms of affinity scheduling. Alan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Aug 13 3:55:47 1999 Delivered-To: freebsd-smp@freebsd.org Received: from overcee.netplex.com.au (overcee.netplex.com.au [202.12.86.7]) by hub.freebsd.org (Postfix) with ESMTP id 5A99514E77 for ; Fri, 13 Aug 1999 03:55:42 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id B09681C1E; Fri, 13 Aug 1999 18:55:16 +0800 (WST) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.0.2 2/24/98 To: Alan Cox Cc: "Josep Maria M. Blanquer" , freebsd-smp@freebsd.org Subject: Re: Questions In-reply-to: Your message of "Thu, 12 Aug 1999 18:17:00 EST." <19990812181700.B27359@nonpc.cs.rice.edu> Date: Fri, 13 Aug 1999 18:55:16 +0800 From: Peter Wemm Message-Id: <19990813105516.B09681C1E@overcee.netplex.com.au> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Alan Cox wrote: > On Wed, Aug 11, 1999 at 08:01:14AM -0700, Josep Maria M. Blanquer wrote: > > ... > > > > 2- The other question is the use of lastcpu field on proc structure: > > I've seen that the field is being updated on cpu_switch() but > > I couldn't find any place where the field is being used. Is this > > a pending work? I mean there is or has been any work on SMP > > specific scheduling? I mean, process balancing, local/global queues > > and so on? > > > > Try asking John Dyson (dyson@iquest.net). I think he has > experimented with some limited forms of affinity scheduling. I've done this BTW and have it currently running. I've turned up a bug or two that look awfully like something is changing p->p_priority of processes while they are on run queues. Even doing trivial affinity makes a big difference here. Trivial meaning that when selecting a process to run, walk the current run queue level and find the first process with a matching lastcpu id rather than just the first in the queue. If no match, then take the head. This is what John did, but I rewrote setrunqueue, remrq in C and moved the process selection out of i386/swtch.s and into C. The compiler generates suprisingly similar code to the assembler version, but when you turn on the U/V pipeline scheduling and the cpu-specific code generation options (eg: use cmove etc) then it seems to do slightly better than the original assembler code. This dramatically simplifies the complexity of cpu_switch() and swtch.s and moves the run queue management to MI code. All that is left in cpu_switch() is the actual context switch code. > Alan Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Aug 13 5: 7:20 1999 Delivered-To: freebsd-smp@freebsd.org Received: from overcee.netplex.com.au (overcee.netplex.com.au [202.12.86.7]) by hub.freebsd.org (Postfix) with ESMTP id EFEFB14D42 for ; Fri, 13 Aug 1999 05:07:11 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id 3E5A51C1E; Fri, 13 Aug 1999 20:06:20 +0800 (WST) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.0.2 2/24/98 Cc: Alan Cox , "Josep Maria M. Blanquer" , freebsd-smp@freebsd.org Subject: Re: Questions In-reply-to: Your message of "Fri, 13 Aug 1999 18:55:16 +0800." <19990813105516.B09681C1E@overcee.netplex.com.au> Date: Fri, 13 Aug 1999 20:06:20 +0800 From: Peter Wemm Message-Id: <19990813120620.3E5A51C1E@overcee.netplex.com.au> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Peter Wemm wrote: > Alan Cox wrote: > > On Wed, Aug 11, 1999 at 08:01:14AM -0700, Josep Maria M. Blanquer wrote: > > > ... > > > > > > 2- The other question is the use of lastcpu field on proc structure: > > > I've seen that the field is being updated on cpu_switch() but > > > I couldn't find any place where the field is being used. Is this > > > a pending work? I mean there is or has been any work on SMP > > > specific scheduling? I mean, process balancing, local/global queues > > > and so on? > > > > > > > Try asking John Dyson (dyson@iquest.net). I think he has > > experimented with some limited forms of affinity scheduling. > > I've done this BTW and have it currently running. [..] This is what John > did, but I rewrote setrunqueue, remrq in C and moved the process selection > out of i386/swtch.s and into C. [..] I forgot to mention.. Mike Smith is the culprit who gave me the idea. It's all his fault. :-) Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Aug 13 13:30: 6 1999 Delivered-To: freebsd-smp@freebsd.org Received: from high-voltage.com (voltage.high-voltage.com [205.243.158.175]) by hub.freebsd.org (Postfix) with SMTP id BCA1214C0F for ; Fri, 13 Aug 1999 13:29:56 -0700 (PDT) (envelope-from BMCGROARTY@high-voltage.com) Date: Fri, 13 Aug 1999 15:11 -0600 From: "Brian McGroarty" To: "Peter Wemm" , "Alan Cox" Cc: "Josep Maria M. Blanquer" , "freebsd-smp" Subject: RE: Re: Questions Message-ID: <6E585390B950D31186D50008C7333C82@high-voltage.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Does this selection mode skew overall processor allocation measurably? I imagine allocation is severely skewed by roundrobin() between schedcpu() calls, but then the high p_estcpu bumps the favored task down a run level queue, compensating with a moment of being starved a processor. The worst case would then be a high priority task with a goodly sized pool of others a few run queue levels down. The process would race and then bob down, float up as p_estcpu is averaged down through schedcpu() calls, soon climb a layer back up for its higher priority, then race and pitch downward again. -----Original Message----- From: Peter Wemm [mailto:peter@netplex.com.au] Sent: Friday, August 13, 1999 4:55 AM To: Brian McGroarty; Alan Cox Cc: Josep Maria M. Blanquer; freebsd-smp Subject: Re: Questions > Try asking John Dyson (dyson@iquest.net). I think he has > experimented with some limited forms of affinity scheduling. I've done this BTW and have it currently running. I've turned up a bug or two that look awfully like something is changing p->p_priority of processes while they are on run queues. Even doing trivial affinity makes a big difference here. Trivial meaning that when selecting a process to run, walk the current run queue level and find the first process with a matching lastcpu id rather than just the first in the queue. If no match, then take the head. This is what John did, but I rewrote setrunqueue, remrq in C and moved the process selection out of i386/swtch.s and into C. The compiler generates suprisingly similar code to the assembler version, but when you turn on the U/V pipeline scheduling and the cpu-specific code generation options (eg: use cmove etc) then it seems to do slightly better than the original assembler code. This dramatically simplifies the complexity of cpu_switch() and swtch.s and moves the run queue management to MI code. All that is left in cpu_switch() is the actual context switch code. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Aug 13 14:10:39 1999 Delivered-To: freebsd-smp@freebsd.org Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (Postfix) with ESMTP id 89F5414DE6 for ; Fri, 13 Aug 1999 14:10:36 -0700 (PDT) (envelope-from tlambert@usr01.primenet.com) Received: (from daemon@localhost) by smtp03.primenet.com (8.9.3/8.9.3) id OAA04480; Fri, 13 Aug 1999 14:09:10 -0700 (MST) Received: from usr01.primenet.com(206.165.6.201) via SMTP by smtp03.primenet.com, id smtpdAAA2da4Qi; Fri Aug 13 14:09:01 1999 Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id OAA18118; Fri, 13 Aug 1999 14:08:59 -0700 (MST) From: Terry Lambert Message-Id: <199908132108.OAA18118@usr01.primenet.com> Subject: Re: Re: Questions To: BMCGROARTY@high-voltage.com (Brian McGroarty) Date: Fri, 13 Aug 1999 21:08:58 +0000 (GMT) Cc: peter@netplex.com.au, alc@cs.rice.edu, blanquer@cs.ucsb.edu, freebsd-smp@FreeBSD.ORG In-Reply-To: <6E585390B950D31186D50008C7333C82@high-voltage.com> from "Brian McGroarty" at Aug 13, 99 03:11:00 pm X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Brian McGroarty writes: > Peter Wemm writes: > > > Try asking John Dyson (dyson@iquest.net). I think he has > > > experimented with some limited forms of affinity scheduling. > > > > I've done this BTW and have it currently running. I've turned up a bug or > > two that look awfully like something is changing p->p_priority of processes > > while they are on run queues. > > > > Even doing trivial affinity makes a big difference here. Trivial meaning > > that when selecting a process to run, walk the current run queue level and > > find the first process with a matching lastcpu id rather than just the > > first in the queue. If no match, then take the head. This is what John > > did, but I rewrote setrunqueue, remrq in C and moved the process selection > > out of i386/swtch.s and into C. The compiler generates suprisingly similar > > code to the assembler version, but when you turn on the U/V pipeline > > scheduling and the cpu-specific code generation options (eg: use cmove etc) > > then it seems to do slightly better than the original assembler code. > > > > This dramatically simplifies the complexity of cpu_switch() and swtch.s > > and moves the run queue management to MI code. All that is left in > > cpu_switch() is the actual context switch code. > > Does this selection mode skew overall processor allocation measurably? > > I imagine allocation is severely skewed by roundrobin() between schedcpu() > calls, but then the high p_estcpu bumps the favored task down a run level > queue, compensating with a moment of being starved a processor. > > The worst case would then be a high priority task with a goodly sized pool of > others a few run queue levels down. The process would race and then bob down, > float up as p_estcpu is averaged down through schedcpu() calls, soon climb a > layer back up for its higher priority, then race and pitch downward again. Yes, this is a highly suboptimal method of achieving affinity. A more correct mechanism would use per CPU ready-to-run queues. Such a method is also amenable to getting multiple processors into user space within a single process, using a slightly modified user space threads scheduler (i.e. one with one threads scheduler stack per expected CPU entry into the user space). The other thing this buys you is the ability to migrate processes between CPU's intentionally, with only reference to a per queue spin lock, rather than use of the Big Giant Lock(tm), which increases concurrency, and, for more than 2 CPU's, can only result in a single stall instead of a global stall, in the kernel scheduler. You still have to write some extra code to monitor instantaneous load over time so you can decide when to migrate, and you have to keep a last_cpu around to make the decision about which run queue to reinsert on after a wakeup, but that's pretty trivial. Also note that Peter's implementation is technically only machine independent for SMP systems that implement a MESI coherency model; this means it's unsuitable for most non-Intel architectures (e.g. the PPC 603 dual processor boxes that remove the L2 cache and use the cache control lines in order to implement only MEI coherency). But this is true for most of the SMP implementation, e.g. there are no hooks in the VM for signalling or cache flushing on systems that don't support automatic cache writeback between processors. Note that such systems are intrinsically less likely to cache bust, when coherency is ensured only when necessary, and are likely to be able to scale to a (potentially much) larger number of processors than the Intel multiprocessing design. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Aug 13 14:54: 1 1999 Delivered-To: freebsd-smp@freebsd.org Received: from overcee.netplex.com.au (overcee.netplex.com.au [202.12.86.7]) by hub.freebsd.org (Postfix) with ESMTP id 9E59C14C20 for ; Fri, 13 Aug 1999 14:53:55 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id 717011C1E; Sat, 14 Aug 1999 05:51:17 +0800 (WST) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.0.2 2/24/98 To: "Brian McGroarty" Cc: "Alan Cox" , "Josep Maria M. Blanquer" , "freebsd-smp" Subject: Re: Questions In-reply-to: Your message of "Fri, 13 Aug 1999 15:11:00 CST." <6E585390B950D31186D50008C7333C82@high-voltage.com> Date: Sat, 14 Aug 1999 05:51:17 +0800 From: Peter Wemm Message-Id: <19990813215117.717011C1E@overcee.netplex.com.au> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org "Brian McGroarty" wrote: > Does this selection mode skew overall processor allocation measurably? > > I imagine allocation is severely skewed by roundrobin() between schedcpu() > calls, but then the high p_estcpu bumps the favored task down a run level > queue, compensating with a moment of being starved a processor. > > The worst case would then be a high priority task with a goodly sized pool of > others a few run queue levels down. The process would race and then bob down, > float up as p_estcpu is averaged down through schedcpu() calls, soon climb a > layer back up for its higher priority, then race and pitch downward again. To give a better picture of what's going on: - we have 128 scheduler "priority levels" at present. - these levels are mapped onto 32 run queues. ie: levels 0, 1, 2, 3 would get inserted into run queue 0. 4,5,6,7 go in queue 1, etc. - we keep a queue busy bit which indicates which queues have something. Under the old algorithm, at switch time, we would pick off the process from the head of the highest priority queue and run it. When it's quantum was up, it would be placed at the back of it's runlevel queue. That's it. If we have 3 processes in the priority range 0,1,2,3, then they get even cpu time for the whole second until the next reschedule happens to rebalance things up. Under my initial hack, instead of just taking the leading process from the highest prority queue, it walks the queue instead and takes the first process that was last on the current cpu. When finishing, it gets put on the back of it's queue and selection happens again. As terry says, this is *far* from optimal, but subjectively it seems to work a heck of a lot better than what we have now. The main flaws: - we only reschedule() once every second, which means that if we get 4 processes in the top queue, and 3 were on cpu 0 and one was last on cpu 1, then cpu0 will split itself between 3 processes for a whole second and cpu1 will spend all it's time on the 4th process. - if the 4th process happens to be a nice 19 process that's been kicked up to top priority due to starvation, then it could well run at higher priority than a the three normal processes if it had the whole cpu to itself. This lack of granularity really sucks and hurts a lot when running things like rc5des etc in the background, even on a single cpu system. It's very spectacular when you have a graphical program running. For one second, it runs at full speed. The next second, it runs at half speed as it's sharing the top run queue with rc5des. Then, it's back to full speed again, and so on every second. I made a few tweaks here a while ago to enable speeding up the scheduler that fixed this, but bde said he had other patches to fix it properly. I ended up frying my patch set somehow and never got back to it. When bde committed his changes, it made little difference here. I think it now takes 1/4 of a second for the lower priority process to loose the top priority now. I think the SMP "BETTER_CLOCK" stuff does similar things. Cheers, -Peter To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Aug 13 15:18:55 1999 Delivered-To: freebsd-smp@freebsd.org Received: from vip.consys.com (VIP.ConSys.COM [209.141.107.6]) by hub.freebsd.org (Postfix) with ESMTP id 5182614E8A for ; Fri, 13 Aug 1999 15:18:50 -0700 (PDT) (envelope-from rcarter@pinyon.org) Received: (from pinyon@localhost) by vip.consys.com (8.9.3/8.9.1) id PAA57570 for freebsd-smp@freebsd.org; Fri, 13 Aug 1999 15:19:05 -0700 (MST) Date: Fri, 13 Aug 1999 15:19:05 -0700 (MST) From: "Russell L. Carter" Message-Id: <199908132219.PAA57570@vip.consys.com> To: freebsd-smp@freebsd.org Subject: Odd swap stats in today's current Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Howdy, I updated an SMP box from a month old current today, and now it seems to think that it wants to swap to wd drives. I don't have any wd drives... Other than that it seems to be running fine, but I don't think it's tried to swap yet, that'll be happening soon enough :-) Maybe this is an off-by-one glitch? Cheers, Russell Here's swapinfo, there should be a da1s1b partition also: rcarter@mazatzal:/var/run [158] swapinfo Device 1K-blocks Used Avail Capacity Type /dev/da0s1b 524160 1900 522260 0% Interleaved /dev/wd0b 850457216 849934976 522240 100% Interleaved /dev/wd0a -128 -128 0 100% Interleaved /dev/wd0a 90483584 90483584 0 100% Interleaved Total 941464832 940420332 1044500 100% And here's the fstab: rcarter@mazatzal:/sys/i386/conf [172] cat /etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/da0s1b none swap sw 0 0 /dev/da1s1b none swap sw 0 0 /dev/da0s1a / ufs rw 1 1 /dev/da1s1e /u1 ufs rw 2 2 /dev/da1s1f /u2 ufs rw 2 2 /dev/da0s1f /usr ufs rw 2 2 /dev/da0s1e /var ufs rw 2 2 proc /proc procfs rw 0 0 rcarter@mazatzal:/sys/i386/conf [173] And the relevant pieces of the kernel config: controller isa0 controller pnp0 # PnP support for ISA controller eisa0 controller pci0 controller fdc0 at isa? port IO_FD1 irq 6 drq 2 disk fd0 at fdc0 drive 0 disk fd1 at fdc0 drive 1 #controller wdc0 at isa? port IO_WD1 irq 14 #disk wd0 at wdc0 drive 0 #disk wd1 at wdc0 drive 1 #controller wdc1 at isa? port IO_WD2 irq 15 #disk wd2 at wdc1 drive 0 #disk wd3 at wdc1 drive 1 # ATAPI devices on wdc? #device wcd0 #IDE CD-ROM #device wfd0 #IDE Floppy (e.g. LS-120) #device wst0 #IDE Tape (e.g. Travan) And the dmesg.boot: (no wd drives here, either) Copyright (c) 1992-1999 The FreeBSD Project. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. FreeBSD 4.0-CURRENT #9: Wed Jul 14 19:33:48 MST 1999 rcarter@mazatzal.consys.com:/usr/src/sys/compile/CURRENT Timecounter "i8254" frequency 1193182 Hz CPU: Pentium II/Xeon/Celeron (686-class CPU) Origin = "GenuineIntel" Id = 0x652 Stepping = 2 Features=0x183fbff real memory = 536870912 (524288K bytes) avail memory = 519004160 (506840K bytes) Programming 24 pins in IOAPIC #0 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 0, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 1, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 2, version: 0x00170011, at 0xfec00000 Preloaded elf kernel "kernel" at 0xc02e7000. Pentium Pro MTRR support enabled, default memory type is uncacheable Probing for PnP devices: npx0: on motherboard npx0: INT 16 interface pcib0: on motherboard pci0: on pcib0 chip0: at device 0.0 on pci0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 chip1: at device 7.1 on pci0 chip2: at device 7.2 on pci0 chip3: at device 7.3 on pci0 pcib2: at device 16.0 on pci0 pci2: on pcib2 vx0: <3COM 3C590 Etherlink III PCI> irq 18 at device 6.0 on pci2 utp/aui/bnc[*utp*]: disable 'auto select' with DOS util! address 00:a0:24:c9:f9:63 fxp0: irq 19 at device 17.0 on pci0 fxp0: Ethernet address 00:e0:81:10:29:d1 ahc0: irq 16 at device 18.0 on pci0 ahc0: aic7895 Wide Channel A, SCSI Id=7, 16/255 SCBs ahc1: irq 16 at device 18.1 on pci0 ahc1: aic7895 Wide Channel B, SCSI Id=7, 16/255 SCBs isa0: on motherboard fdc0: at port 0x3f0-0x3f7 irq 6 drq 2 on isa0 fdc0: FIFO enabled, 8 bytes threshold fd0: <1440-KB 3.5" drive> at fdc0 drive 0 atkbdc0: at port 0x60-0x6f on isa0 atkbd0: irq 1 on atkbdc0 vga0: at port 0x3b0-0x3df iomem 0xa0000-0xbffff on isa0 sc0: on isa0 sc0: VGA <16 virtual consoles, flags=0x200> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A sio1: configured irq 3 not in bitmap of probed irqs 0 APIC_IO: Testing 8254 interrupt delivery APIC_IO: routing 8254 via pin 2 vx0 XXX: driver didn't set ifq_maxlen Waiting 15 seconds for SCSI devices to settle SMP: AP CPU #1 Launched! changing root device to da0s1a da1 at ahc1 bus 0 target 6 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 40.000MB/s transfers (20.000MHz, offset 8, 16bit), Tagged Queueing Enabled da1: 4357MB (8925000 512 byte sectors: 255H 63S/T 555C) da0 at ahc0 bus 0 target 3 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit), Tagged Queueing Enabled da0: 4357MB (8925000 512 byte sectors: 255H 63S/T 555C) [end...] To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Aug 13 15:37:58 1999 Delivered-To: freebsd-smp@freebsd.org Received: from high-voltage.com (voltage.high-voltage.com [205.243.158.175]) by hub.freebsd.org (Postfix) with SMTP id 22BF914EE8 for ; Fri, 13 Aug 1999 15:37:32 -0700 (PDT) (envelope-from BMCGROARTY@high-voltage.com) Date: Fri, 13 Aug 1999 17:27 -0600 From: "Brian McGroarty" To: "Peter Wemm" Cc: "Alan Cox" , "Josep Maria M. Blanquer" , "freebsd-smp" Subject: RE: Re: Questions Message-ID: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org From: Peter Wemm [mailto:peter@netplex.com.au] > > This lack of granularity really sucks and hurts a lot when > running things like > rc5des etc in the background, even on a single cpu system. It's very > spectacular when you have a graphical program running. For > one second, it > runs at full speed. The next second, it runs at half speed > as it's sharing > the top run queue with rc5des. Then, it's back to full speed > again, and so > on every second. Yeah, I stopped running rc5des for this reason. The imbalance exists in the single processor model when one process is in a queue above or below a pool of other processes, yes. But your model seems like it has the potential to take a nice 19 process, develop complete affinity on one processor with everything else in a pool below, and then even if it falls into the same pool as the others, they've developed affinity for the other processor. The result? Zoom! nice 19 is shot downward and out of the crowd in the next scheduling interval like the proverbial watermellon seed. Without this affinity model, the process wouldn't likely dip below the pooled process queue and it would level off its processor usage before timidly starting to climb again. I expect your test model approaches a near perfect model as you tighten the scheduling quantum to a 1:1 schedcpu():roundrobin() interval if we ignore the schedcpu(). Hard to do with all that resultant locking, but that's changing, I guess. Just as a thought exercise, I wonder how optimal an incrementally changing schedcpu() could be, as opposed to the current block reorganization. > I made a few tweaks here a while ago to enable speeding up > the scheduler > that fixed this, but bde said he had other patches to fix it > properly. I > ended up frying my patch set somehow and never got back to > it. When bde Oi, that's NOT what I wanted to hear. I'd much rather learn about this in C than Intel assembly. > committed his changes, it made little difference here. I > think it now takes > 1/4 of a second for the lower priority process to loose the > top priority now. > I think the SMP "BETTER_CLOCK" stuff does similar things. I'll look at BETTER_CLOCK. I like Terry's model though - I'm going to do some reading and see if I can find something similar for reference. Do you know of, or have you considered, any other processor affinity scheduling models? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Aug 13 15:54:21 1999 Delivered-To: freebsd-smp@freebsd.org Received: from csvax.cs.caltech.edu (csvax.cs.caltech.edu [131.215.131.131]) by hub.freebsd.org (Postfix) with ESMTP id B5B0E14C13 for ; Fri, 13 Aug 1999 15:54:19 -0700 (PDT) (envelope-from mika@gauguin.cs.caltech.edu) Received: from gauguin.cs.caltech.edu (gauguin.cs.caltech.edu [131.215.78.41]) by csvax.cs.caltech.edu (8.9.1/8.9.1) with ESMTP id PAA05807; Fri, 13 Aug 1999 15:54:25 -0700 (PDT) Received: from localhost.cs.caltech.edu (localhost.cs.caltech.edu [127.0.0.1]) by gauguin.cs.caltech.edu (8.8.7/8.7.3) with SMTP id PAA02257; Fri, 13 Aug 1999 15:54:20 -0700 (PDT) Message-Id: <199908132254.PAA02257@gauguin.cs.caltech.edu> X-Authentication-Warning: gauguin.cs.caltech.edu: localhost.cs.caltech.edu [127.0.0.1] didn't use HELO protocol To: Peter Wemm Cc: alc@cs.rice.edu, blanquer@cs.ucsb.edu, freebsd-smp@FreeBSD.ORG, BMCGROARTY@high-voltage.com, tech-kern@netbsd.org Subject: Re: Questions In-reply-to: Your message of "Sat, 14 Aug 1999 05:51:17 +0800." <19990813215117.717011C1E@overcee.netplex.com.au> Date: Fri, 13 Aug 1999 15:54:20 -0700 From: Mika Nystrom Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Peter Wemm writes: ... > > to top priority due to starvation, then it could well run at higher > priority than a the three normal processes if it had the whole cpu to itself >. > >This lack of granularity really sucks and hurts a lot when running things like >rc5des etc in the background, even on a single cpu system. It's very >spectacular when you have a graphical program running. For one second, it >runs at full speed. The next second, it runs at half speed as it's sharing >the top run queue with rc5des. Then, it's back to full speed again, and so >on every second. > >I made a few tweaks here a while ago to enable speeding up the scheduler >that fixed this, but bde said he had other patches to fix it properly. I >ended up frying my patch set somehow and never got back to it. When bde >committed his changes, it made little difference here. I think it now takes >1/4 of a second for the lower priority process to loose the top priority now. >I think the SMP "BETTER_CLOCK" stuff does similar things. > ... Hello, We have a user here who insists on running rc5des on a lab full of NetBSD/i386 machines (all UP, of course). I made a simple hack to the scheduler to solve this problem. The scheduler on our machines simply reserves the bottom-most run queue for nice 20 processes. I PR'd it for NetBSD, but it's such a crude hack that I'm not surprised it hasn't gotten any attention. Well, it's crude, but it really does do the trick as far as these rc5 clients are concerned. You simply cannot tell that one is running in the background. I was going to apply the patch to FreeBSD but FreeBSD has code that does exactly the same thing using the rtprio mechanism.. I would guess that using idprio on the FreeBSD des clients would work the same as my NetBSD hack. Of course, processor affinity, etc. is a much bigger question that this two-liner does nothing to address. Mika void resetpriority(p) register struct proc *p; { register unsigned int newpriority; newpriority = PUSER + p->p_estcpu / 4 + 2 * (p->p_nice - NZERO); #ifdef HARDNICE newpriority = min(newpriority, MAXPRI-PPQ-1); if (p->p_nice == (PRIO_MAX + NZERO) ) newpriority = MAXPRI; #else newpriority = min(newpriority, MAXPRI); #endif p->p_usrpri = newpriority; if (newpriority < curpriority) need_resched(); } To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Aug 14 0: 2:46 1999 Delivered-To: freebsd-smp@freebsd.org Received: from godzilla.zeta.org.au (godzilla.zeta.org.au [203.26.10.9]) by hub.freebsd.org (Postfix) with ESMTP id A0E0514D2C for ; Sat, 14 Aug 1999 00:02:37 -0700 (PDT) (envelope-from bde@godzilla.zeta.org.au) Received: (from bde@localhost) by godzilla.zeta.org.au (8.8.7/8.8.7) id RAA16805; Sat, 14 Aug 1999 17:02:35 +1000 Date: Sat, 14 Aug 1999 17:02:35 +1000 From: Bruce Evans Message-Id: <199908140702.RAA16805@godzilla.zeta.org.au> To: BMCGROARTY@high-voltage.com, peter@netplex.com.au Subject: Re: Questions Cc: alc@cs.rice.edu, blanquer@cs.ucsb.edu, freebsd-smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org >This lack of granularity really sucks and hurts a lot when running things like >rc5des etc in the background, even on a single cpu system. It's very >spectacular when you have a graphical program running. For one second, it >runs at full speed. The next second, it runs at half speed as it's sharing >the top run queue with rc5des. Then, it's back to full speed again, and so >on every second. > >I made a few tweaks here a while ago to enable speeding up the scheduler >that fixed this, but bde said he had other patches to fix it properly. I Other patches to fix other scheduling bugs. I was still looking for the major bug that let rc5des (or any process) run in preference to higher priority processes on systems with shared PCI, EISA) irqs. >ended up frying my patch set somehow and never got back to it. When bde >committed his changes, it made little difference here. I think it now takes I never got back to committing my other patches. >1/4 of a second for the lower priority process to loose the top priority now. >I think the SMP "BETTER_CLOCK" stuff does similar things. statclock() calls resetpriority() for every 4 user-mode calls, i.e., every 1/32 seconds while user processes are active. Perhaps it takes 8 calls for the changes to affect the run queues (maybe 4 for the priority granularity and 1 for a bug that I haven't committed the fix for yet?). 1/4 second isn't as long as it used to be :). The magic mask of 3 in statclock() should depend on the frequency of statclock(). Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Aug 14 2:55:50 1999 Delivered-To: freebsd-smp@freebsd.org Received: from overcee.netplex.com.au (overcee.netplex.com.au [202.12.86.7]) by hub.freebsd.org (Postfix) with ESMTP id E79EF14DF7 for ; Sat, 14 Aug 1999 02:55:41 -0700 (PDT) (envelope-from peter@netplex.com.au) Received: from netplex.com.au (localhost [127.0.0.1]) by overcee.netplex.com.au (Postfix) with ESMTP id 125FE1C1E for ; Sat, 14 Aug 1999 17:55:56 +0800 (WST) (envelope-from peter@netplex.com.au) X-Mailer: exmh version 2.0.2 2/24/98 To: smp@freebsd.org Subject: simple affinity and pagezero etc. Date: Sat, 14 Aug 1999 17:55:56 +0800 From: Peter Wemm Message-Id: <19990814095556.125FE1C1E@overcee.netplex.com.au> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org As part of the tinkering I've been doing with cpu_switch() etc, I decided to have another shot at John Dyson's idea of making the idle loop pagezero stuff into an idleprio kthread/process/whatever. Anyway, it seems to simplify things a lot, but it's caused a side effect I hadn't considered.. On this system I run the rc5des client, which rfork()'s into two threads at nice 19. These effectively block out the process entirely. On the system with these patches and soft affinity: last pid: 1380; load averages: 2.94, 2.96, 2.98 up 0+10:14:12 17:39:59 CPU states: 0.4% user, 99.2% nice, 0.0% system, 0.4% interrupt, 0.0% idle PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 1158 peter 105 20 824K 452K RUN 1 60:33 97.95% 97.95% rc564 1157 peter 105 20 824K 452K CPU0 0 60:35 96.92% 96.92% rc564 1380 root 45 0 1560K 764K CPU1 0 0:00 2.31% 0.68% top 4 root 18 52 0K 0K RUN 1 0:03 0.00% 0.00% pgzero Anyway, the side effect that I hadn't considered was that the pgzero process sits in the run queue permanently and skews the load average up by one. ie: the load average is at 3 even though there are two running processes. On the other hand, it does simplify things a lot and I'm far more confident that it's "robust". I'm tempted to move it from max idprio priority to nice 19 so that it isn't starved for so long. I suspect it's probably a little better do occasionally do some pagezeroing if we've left it for a for long enough for the scheduler to raise it's priority enough to get a shot at the cpu for a moment. Cheers, -Peter To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Aug 14 3:13:24 1999 Delivered-To: freebsd-smp@freebsd.org Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.40.131]) by hub.freebsd.org (Postfix) with ESMTP id 20DBB1510C for ; Sat, 14 Aug 1999 03:13:02 -0700 (PDT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.9.3/8.9.2) with ESMTP id MAA10669; Sat, 14 Aug 1999 12:12:49 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Peter Wemm Cc: smp@FreeBSD.ORG Subject: Re: simple affinity and pagezero etc. In-reply-to: Your message of "Sat, 14 Aug 1999 17:55:56 +0800." <19990814095556.125FE1C1E@overcee.netplex.com.au> Date: Sat, 14 Aug 1999 12:12:49 +0200 Message-ID: <10667.934625569@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In message <19990814095556.125FE1C1E@overcee.netplex.com.au>, Peter Wemm writes: >As part of the tinkering I've been doing with cpu_switch() etc, I decided >to have another shot at John Dyson's idea of making the idle loop pagezero >stuff into an idleprio kthread/process/whatever. > >[...] > >I'm tempted to move it from max idprio priority to nice 19 so that it isn't >starved for so long. makes sense. -- Poul-Henning Kamp FreeBSD coreteam member phk@FreeBSD.ORG "Real hackers run -current on their laptop." FreeBSD -- It will take a long time before progress goes too far! To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Aug 14 3:57: 8 1999 Delivered-To: freebsd-smp@freebsd.org Received: from godzilla.zeta.org.au (godzilla.zeta.org.au [203.26.10.9]) by hub.freebsd.org (Postfix) with ESMTP id 3835214DD2 for ; Sat, 14 Aug 1999 03:57:02 -0700 (PDT) (envelope-from bde@godzilla.zeta.org.au) Received: (from bde@localhost) by godzilla.zeta.org.au (8.8.7/8.8.7) id UAA30740; Sat, 14 Aug 1999 20:54:58 +1000 Date: Sat, 14 Aug 1999 20:54:58 +1000 From: Bruce Evans Message-Id: <199908141054.UAA30740@godzilla.zeta.org.au> To: peter@netplex.com.au, smp@FreeBSD.ORG Subject: Re: simple affinity and pagezero etc. Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org >As part of the tinkering I've been doing with cpu_switch() etc, I decided >to have another shot at John Dyson's idea of making the idle loop pagezero >stuff into an idleprio kthread/process/whatever. I don't think we would want to always use it, since it would disturb caches and increase latency for restarting user processes. Perhaps it should only start up after the system has been idle for a while (this means 1/HZ or 2/HZ in practice unless idle() spins testing the time, and 10-20 ms may be good in theory too because it is similar to disk seek times). Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Aug 14 16:36:50 1999 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [209.157.86.2]) by hub.freebsd.org (Postfix) with ESMTP id E51631516B for ; Sat, 14 Aug 1999 16:36:47 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.9.3/8.9.1) id QAA98194; Sat, 14 Aug 1999 16:34:01 -0700 (PDT) (envelope-from dillon) Date: Sat, 14 Aug 1999 16:34:01 -0700 (PDT) From: Matthew Dillon Message-Id: <199908142334.QAA98194@apollo.backplane.com> To: "Russell L. Carter" Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Odd swap stats in today's current References: <199908132219.PAA57570@vip.consys.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org :Howdy, : :I updated an SMP box from a month old current today, and now it :seems to think that it wants to swap to wd drives. : :I don't have any wd drives... Other than that it seems to be :running fine, but I don't think it's tried to swap yet, that'll :be happening soon enough :-) Maybe this is an off-by-one glitch? : :Cheers, :Russell : :Here's swapinfo, there should be a da1s1b partition also: If you didn't buildworld then I recommend rebuilding libkvm and pstat. Also make sure your includes are up to date. cd /usr/src make includes cd /usr/src/lib/libkvm make make install make clean cd /usr/src/usr.sbin/pstat make make install make clean rm /var/db/kvm_kernel.db kvm_mkdb See if that helps. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message