From owner-freebsd-smp Mon Dec 14 11:45:55 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id LAA01600 for freebsd-smp-outgoing; Mon, 14 Dec 1998 11:45:55 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from post.mail.demon.net (post-12.mail.demon.net [194.217.242.41]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id LAA01569; Mon, 14 Dec 1998 11:45:24 -0800 (PST) (envelope-from james@westongold.com) Received: from [158.152.96.124] (helo=wgp01.wgold.demon.co.uk) by post.mail.demon.net with esmtp (Exim 2.054 #1) id 0zpdvh-0007In-00; Mon, 14 Dec 1998 19:45:18 +0000 Received: by WGP01 with Internet Mail Service (5.5.1960.3) id ; Mon, 14 Dec 1998 19:27:53 -0000 Message-ID: <32BABEF63EAED111B2C5204C4F4F5020183A@WGP01> From: James Mansion To: Terry Lambert , mal@algonet.se Cc: alk@pobox.com, peter@netplex.com.au, gpalmer@FreeBSD.ORG, marcelk@stack.nl, smp@FreeBSD.ORG Subject: RE: Pthreads and SMP Date: Mon, 14 Dec 1998 19:27:43 -0000 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.1960.3) Content-Type: text/plain Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > -----Original Message----- > From: Terry Lambert [mailto:tlambert@primenet.com] > Sent: Friday, December 11, 1998 1:51 AM >... > > Are you saying that processes migrating randomly between > processors, > > cache busting as they go, is less of a problem for a given parallell > > application than kernel threads doing the same? Or that > the scheduler > > is able to keep processes mostly on the same processor but would not > > easily be made able to do the same for kernel threads? > > The latter, and not on purpose, but merely as an artifact of how > the scheduler works. > > You can get process migration, but it just doesn't happen very often > in practice. > > The use of a (mostly) shared VM space between kernel threads is a > major source of cache line invalidation/update and IPI's between the > processors in an SMP system, and this adds to the problem. Seperate > processes would not require invalidation,/update since the address > space is only in one cache at a time, not two. So, in what way is an application built from multiple processes with shared state in a shared memory segment going to be better than an application with multiple kernel threads running in a largely shared address space? Is this simply an artifact of the heap manager being an additional shared data structure that is updated? If the threads run (largely) in thread-specific heaps, and/or the main heap is very thread-aware (likeSmartHeap for SMP, for example) where's the beef? It seems this isn't a complaint about kernel threads per se, but about the negative effects of having much state that is updated by multiple threads and also read frequently, which is a design issue for the MT application. And one that is there for an app built from multiple processes with explicit shared memory, too. James To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 14 23:33:37 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id XAA25902 for freebsd-smp-outgoing; Mon, 14 Dec 1998 23:33:37 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id XAA25877; Mon, 14 Dec 1998 23:33:31 -0800 (PST) (envelope-from tlambert@usr06.primenet.com) Received: (from daemon@localhost) by smtp03.primenet.com (8.8.8/8.8.8) id AAA23118; Tue, 15 Dec 1998 00:33:25 -0700 (MST) Received: from usr06.primenet.com(206.165.6.206) via SMTP by smtp03.primenet.com, id smtpd023023; Tue Dec 15 00:33:17 1998 Received: (from tlambert@localhost) by usr06.primenet.com (8.8.5/8.8.5) id AAA12768; Tue, 15 Dec 1998 00:32:54 -0700 (MST) From: Terry Lambert Message-Id: <199812150732.AAA12768@usr06.primenet.com> Subject: Re: Pthreads and SMP To: james@westongold.com (James Mansion) Date: Tue, 15 Dec 1998 07:32:54 +0000 (GMT) Cc: tlambert@primenet.com, mal@algonet.se, alk@pobox.com, peter@netplex.com.au, gpalmer@FreeBSD.ORG, marcelk@stack.nl, smp@FreeBSD.ORG In-Reply-To: <32BABEF63EAED111B2C5204C4F4F5020183A@WGP01> from "James Mansion" at Dec 14, 98 07:27:43 pm X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > So, in what way is an application built from multiple processes with > shared state in a shared memory segment going to be better than an > application with multiple kernel threads running in a largely shared > address space? I don't understand. Why do you need to share state? If you need to share state, then why do you need threads instead of a select-driven finite state automaton? I guess when all you have is a threads implementation, everything looks like a thread... For a shared context server, the viability of the server is based on how *little* context you actually have to contend for between processors. One of the main tasks we had to do when designing the shared context work-to-do architecture for the NWU (NetWare for UNIX) product at Novell was in minimization of collision domains. It was that, and the need to contend for resources (mostly quantum) that drove the decision to *not* use SVR4 style threads, but to use multiple processes and place the context in a shared memory segment instead. There were also issues that this could address that threading could not; specifically, "hot engine scheduling". In that scenario, the streams MUX handed out work to do in LIFO order of calling process with work to do. The effect we were looking for was to optimize for the most data pages in core for a given process (thread/engine) when work needed to be done. The load rarely cause CPU time to accumulate on more than 5 engines simultaneously, even serving 256 client PC's simultaneously in "packet burst" mode. > Is this simply an artifact of the heap manager being an additional > shared data structure that is updated? If all the address space is shared, then an invalidation of an area in the address space by a thread on CPU #5 makes updates to the other CPU's, even though the threads running on those CPU's may never actually use the data. It's more an artifact of having a common page table between all of the processors, such that the same invalidations affect all of the processors instead of one processor. I guess you could call this heap, but it's more of a PTDE issue, if you get down to it. > If the threads run (largely) in thread-specific heaps, and/or the > main heap is very thread-aware (likeSmartHeap for SMP, for example) > where's the beef? On and around cache-line boundaries actually. You pretty much want them one or more cache lines apart. On many modern Intel processors, this is 64k -- 16 pages. You can pretty much guarantee this statistically by using diffterent mappings -- different processes -- to implement your contexts for your work-to-do engines. > It seems this isn't a complaint about kernel threads per se, but about > the negative effects of having much state that is updated by multiple > threads and also read frequently, which is a design issue for the MT > application. And one that is there for an app built from multiple > processes with explicit shared memory, too. Yes, it's a problem when you share context between threads, no matter how you do it. The moral to the story is "don't do that". If a problem is capable of being parallelized, then it makes sense to try and compute it in parallel; if it isn't, then no amount of adding PC's together is going to get you a virtual supercomputer. The problem space mappable using SMP is not trivially small, but it's a hell of a lot smaller than the problem space that can be mapped with 32 times the computations per clock cycle on a linear uniprocessor. Just because you have a lot of work to do doesn't mean using threads will make it faster. Or SMP, for that matter. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 15 05:31:26 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id FAA27874 for freebsd-smp-outgoing; Tue, 15 Dec 1998 05:31:26 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from databus.databus.com (databus.databus.com [198.186.154.34]) by hub.freebsd.org (8.8.8/8.8.8) with SMTP id FAA27868 for ; Tue, 15 Dec 1998 05:31:24 -0800 (PST) (envelope-from barney@databus.databus.com) From: Barney Wolff To: smp@FreeBSD.ORG Date: Tue, 15 Dec 1998 08:20 EST Subject: Re: Pthreads and SMP Content-Type: text/plain Message-ID: <367664a50.5e3e@databus.databus.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org The debate has crossed the border into silliness. The issue is not whether threads are good or bad, but simply whether they will be available to designers. If I can't use Posix threads to keep multiple cpus busy from a single address space, I'm not interested in FreeBSD as an SMP OS. Yes, I can and do use multiple processes and select, often. But if I can't have a decent pthread implementation, FreeBSD is taking itself out of the running. For each problem, I'll choose the technique that seems best, but I'll choose it on an OS that leaves the choice to me. Barney Wolff To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 15 07:40:38 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id HAA09939 for freebsd-smp-outgoing; Tue, 15 Dec 1998 07:40:38 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from alpha.netaccess.on.ca (netaccess.on.ca [199.243.225.10]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id HAA09930; Tue, 15 Dec 1998 07:40:34 -0800 (PST) (envelope-from rob@ControlQ.com) Received: from fatlady.controlq.com (dial057.netaccess.on.ca [199.243.225.185]) by alpha.netaccess.on.ca (8.8.5/8.7.3) with SMTP id KAA13453; Tue, 15 Dec 1998 10:39:20 -0500 (EST) Date: Tue, 15 Dec 1998 10:44:53 -0500 (EST) From: "Robert S. Sciuk" To: Terry Lambert cc: James Mansion , mal@algonet.se, alk@pobox.com, peter@netplex.com.au, gpalmer@FreeBSD.ORG, marcelk@stack.nl, smp@FreeBSD.ORG Subject: Re: Pthreads and SMP In-Reply-To: <199812150732.AAA12768@usr06.primenet.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, 15 Dec 1998, Terry Lambert wrote: much relevant experience and keen observation snipped ... (hmmm a `yes but'..) > > Just because you have a lot of work to do doesn't mean using threads > will make it faster. Or SMP, for that matter. > Terry, I agree with your above statement, and I generally find your posts to be well though and useful. I must take the side of the pthreads people on this thread, however. Your experience with NWU is valid -- and probably close to an optimal solution for high performance daemons in the Unix context. The schism here, IMHO, seems to be between those who have embraced the pthreads approach, vs those who might be capable of implementing kernel scheduling for the pthreads API's. Processor affinity and cache issues are not immediately apparent to those who have not had to squeeze the last bit of performance out of system level software (as you have). I argue that most applications can be successful, however, without completely linear scalability. The lack of a consistent lightweight lock and synchronization mechanism (eg: user space memory semaphores rather than the heavy kernel SYSV sems) across platforms makes shared (mmap) memory a much harder implementation than pthreads for the average programmer. The select() mechanism and asynch io is also a viable, albeit conceptually more difficult to implement for the average programmer. Pthreads offer the potential for an application level developer type to harness multiple CPU architectures for their own applications in a pseudo portable fashion. CPU affinity and scheduling notwithstanding, it is conceptually simpler to implement a threaded application `correctly'. Reading between the lines, you seem to be saying that writing good (scalable?) applications is hard, and should be hard. Pthreads is a way of allowing at least some of the power of SMP boxes to be accessed by a larger number of developers than some of the historical approaches we have used. (By the way, I was an OEM for NWU on the HP platforms for some years in my former life where the ISO model of the network model was, if not coined, at least widely espoused. I think the top three layers are somewhat universal 8-). Philosopy Religion Politics Application Presentation System Transport Network Datalink Physical I hope that this rather rambling diatribe will be taken in the spirit that it is offered. Cheers, Rob. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Robert S. Sciuk 1032 Howard Rd. PO Box 6A Ph:905 632-2466 Control-Q Research Burlington, Ont. Canada Fx:905 632-7417 rob@ControlQ.com L7R 3X5 http://www.ControlQ.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 15 08:33:01 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id IAA15799 for freebsd-smp-outgoing; Tue, 15 Dec 1998 08:33:01 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from fast.cs.utah.edu (fast.cs.utah.edu [155.99.212.1]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id IAA15784 for ; Tue, 15 Dec 1998 08:32:56 -0800 (PST) (envelope-from vanmaren@fast.cs.utah.edu) Received: (from vanmaren@localhost) by fast.cs.utah.edu (8.9.1/8.9.1) id JAA26636 for smp@FreeBSD.ORG; Tue, 15 Dec 1998 09:32:51 -0700 (MST) Date: Tue, 15 Dec 1998 09:32:51 -0700 (MST) From: Kevin Van Maren Message-Id: <199812151632.JAA26636@fast.cs.utah.edu> To: smp@FreeBSD.ORG Subject: Re: Pthreads and SMP Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Sometimes I have to remember to take what Terry says with a grain of salt. Modern small-scale SMPS (2 or 4-way) have very fast and efficient communication. Yes, the less communication between threads the faster they will run. Yes, CPU-inheritance scheduling can make a big difference, especially with large per-cpu caches. Yes, NT doesn't scale very well on the 4 and 8-way hardware because of it's synchronization mechanisms. That aside, it is most certainly desirable to be able to run multiple threads in parallel. The extent to which user threads are mapped onto processors is best controlled by some provided mechanism (such as pthread_setconcurrency and pthread_getconcurrency) rather than an inflexible policy such as "I believe it may be slow to run multiple threads at the same time". As for Terry's beef about the page table, I don't know how often a typical app gets its page table updated, but I wouldn't think that would be common except when a) you are paging and other performance penalties are likely to be in the noise or b) more memory is being allocated/accessed by the process. It is only necessary to do a TLB-shootdown when restricting the mappings. It isn't a problems if a processor takes a trap because its TLB was out of date and the page is really valid; it simply loads the new info and continues. I believe we should add the necessary mechanisms to run threads in parallel and THEN look at the actual performance problems and address them. If that means the scheduler needs to be improved, fine, we improve the scheduler. If some applications run slower on multiple processors, we just have them call pthread_setconcurrency(1). Shoot, Terry can make the default to be 1 on his machines. Personally, I would like to be able to use pthread_create() instead of fork() to handle computation- bound requests. Terry has a point about wanting to design the system to be fast from the beginning. That is almost certainly better than to design something, realize it is way to slow, and then hack on it forever. However, having this working in the short term and rewriting it for the long term doesn't upset me too much -- I just want it working, and it will certainly be good enough for a large range of applications (even it it isn't large enough or good enough for Terry). Kevin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 15 09:54:15 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id JAA27511 for freebsd-smp-outgoing; Tue, 15 Dec 1998 09:54:15 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from fleming.cs.strath.ac.uk (fleming.cs.strath.ac.uk [130.159.196.126]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id JAA27415; Tue, 15 Dec 1998 09:53:56 -0800 (PST) (envelope-from roger@cs.strath.ac.uk) Received: from cs.strath.ac.uk (posh.dmem.strath.ac.uk [130.159.202.3]) by fleming.cs.strath.ac.uk (8.8.8/8.8.8) with ESMTP id RAA17503 Tue, 15 Dec 1998 17:53:13 GMT Message-ID: <3676A1F2.5D5654AD@cs.strath.ac.uk> Date: Tue, 15 Dec 1998 17:52:50 +0000 From: Roger Hardiman Organization: Strathclyde Uni X-Mailer: Mozilla 4.05 [en] (X11; I; FreeBSD 3.0-RELEASE i386) MIME-Version: 1.0 To: nate@mt.sri.com, mobile@FreeBSD.ORG, smp@FreeBSD.ORG Subject: SMP and PCMCIA give Kernel Panic - fix found Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, I now have a patch which will get PCMCIA cards working on SMP kernels. The problem was the differences in the IRQ allocation with APIC IO. Can you all try it out for me and see how you get on. Then, I'll need someone to commit the patch for me. Any offers? Rather than do a patch, I've stuck a new version of pcic.c on ftp://telepresence.dmem.strath.ac.uk/pub/pcic/pcic.c This is a modified version of pcic.c from 3.0-RELEASE which can be found at ftp://telepresence.dmem.strath.ac.uk/pub/pcic/pcic.c.original (so you can generate your own diffs) Bye Roger To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 15 17:47:16 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id RAA02779 for freebsd-smp-outgoing; Tue, 15 Dec 1998 17:47:16 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from Chuska.ConSys.COM (Chuska.ConSys.COM [209.141.107.5]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id RAA02774 for ; Tue, 15 Dec 1998 17:47:14 -0800 (PST) (envelope-from rcarter@psf.Pinyon.ORG) Received: from psf.Pinyon.ORG (ip-17-196.prc.primenet.com [207.218.17.196]) by Chuska.ConSys.COM (8.9.1/8.9.1) with ESMTP id SAA17893; Tue, 15 Dec 1998 18:47:07 -0700 (MST) Received: from psf.Pinyon.ORG (localhost [127.0.0.1]) by psf.Pinyon.ORG (8.9.1/8.8.7) with ESMTP id SAA03907; Tue, 15 Dec 1998 18:44:15 -0700 (MST) Message-Id: <199812160144.SAA03907@psf.Pinyon.ORG> X-Mailer: exmh version 2.0.2 2/24/98 To: Terry Lambert cc: smp@FreeBSD.ORG Subject: Re: Pthreads and SMP In-reply-to: Your message of "Tue, 15 Dec 1998 07:32:54 GMT." <199812150732.AAA12768@usr06.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 15 Dec 1998 18:44:14 -0700 From: "Russell L. Carter" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > The problem space mappable using SMP is not trivially small, but > it's a hell of a lot smaller than the problem space that can be > mapped with 32 times the computations per clock cycle on a linear > uniprocessor. . I've absorbed (maybe reflected is more accurate) considerable netnoise flack from pointing out that the problem space for distributed memory parallel systems is considerably (even exponentially) smaller than that for SMP. The reason is the same as for SMP vs. uniprocessor. If you are unfortunate to be living in the real world of memory with hierarchies of different access speeds and don't have nearly complete locality for memory references for each and every parallelizable task you are (nearly always) hosed. Wait a minute! Did I just agree with Terry? :-) Russell > > Terry Lambert > terry@lambert.org > --- > Any opinions in this posting are my own and not those of my present > or previous employers. > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 15 19:05:02 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id TAA10714 for freebsd-smp-outgoing; Tue, 15 Dec 1998 19:05:02 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id TAA10695 for ; Tue, 15 Dec 1998 19:05:00 -0800 (PST) (envelope-from tlambert@usr05.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.8.8/8.8.8) id UAA08429; Tue, 15 Dec 1998 20:04:51 -0700 (MST) Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp02.primenet.com, id smtpd008360; Tue Dec 15 20:04:43 1998 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id UAA13431; Tue, 15 Dec 1998 20:04:31 -0700 (MST) From: Terry Lambert Message-Id: <199812160304.UAA13431@usr05.primenet.com> Subject: Re: Pthreads and SMP To: vanmaren@fast.cs.utah.edu (Kevin Van Maren) Date: Wed, 16 Dec 1998 03:04:31 +0000 (GMT) Cc: smp@FreeBSD.ORG In-Reply-To: <199812151632.JAA26636@fast.cs.utah.edu> from "Kevin Van Maren" at Dec 15, 98 09:32:51 am X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > That aside, it is most certainly desirable to be able to run > multiple threads in parallel. The extent to which user threads > are mapped onto processors is best controlled by some provided > mechanism (such as pthread_setconcurrency and pthread_getconcurrency) > rather than an inflexible policy such as "I believe it may be > slow to run multiple threads at the same time". These two interfaces are optional in a conforming pthreads. > As for Terry's beef about the page table, I don't know how often > a typical app gets its page table updated, but I wouldn't think > that would be common except when a) you are paging and other > performance penalties are likely to be in the noise or b) more > memory is being allocated/accessed by the process. It is only > necessary to do a TLB-shootdown when restricting the mappings. > It isn't a problems if a processor takes a trap because its TLB > was out of date and the page is really valid; it simply loads > the new info and continues. The problem is write faults on pages that are in the page tables of both processors because there is only one page table for all of the threads in a single process. This means that you need to invalidate or update cache contents on CPUs athat are, in fact, not *using* the cache contents and could care less. This typically happens in copy on write faults and on stack growth when you hit a guard page, especially if you are passing the addresses of auto variable between threads. It's my personal experience that people use threads because they don't know how to program effectively, and using threads efficiently requires a lot of saftey harness code in the OS for these people to actually gain the benefit they think they will gain. That said, all I was trying to point out are that there are constraints on the efficiency of kernel threads that no one has addressed up to this point except Sun Microsystems, and even then, I think they screwed up the quantum model pretty badly (if the scheduler gives me a quantum, it's *my* damn quantum, and if the scheduler will take it away from me for making a blocking system call, then *screw* the scheduler, I won't make blocking calls). The name of the game is to minimize context switch overhead. > I believe we should add the necessary mechanisms to run threads > in parallel and THEN look at the actual performance problems > and address them. These mechanisms already exist, as has been pointed out countless times on this list. They just aren't packaged up with their glue code in a nice "pthread_create" routine some place, because doing that without further kernel support would result in abysmal performance, and would, in fact, be counter-productive. The main reason for the poor performance are the issues I've outlined here. You can very easily go to the -current list archives and search for "John Dyson" and get a copy of the glue code. Or you could directly ask jmb@freebsd.org for the code. > If that means the scheduler needs to be improved, fine, we improve > the scheduler. If some applications run slower on multiple > processors, we just have them call pthread_setconcurrency(1). Shoot, > Terry can make the default to be 1 on his machines. Personally, I > would like to be able to use pthread_create() instead of fork() to > handle computation-bound requests. Then feel free to integrate John's vfork based kernel threading into your libc_r, and to add the appropriate pthread_setconcurrency() functions to bring the implementation up to some documented standard, instead of teh limbo between Draft 4 and Draft 10 where it currently lives. There's really nothing stopping you from using the code; it was posted to the list. It's just that it would be real silly to abandon a working Draft 10 (standard) pthreads to chase after what some people in this thread are claiming is a computational holy grail, and which others in this thread have already had experienvce with on Solaris 2.3 and below and SVR4.0.2 and UnixWare 2.x. I can tell you: it's not even a grail-shaped beacon. > Terry has a point about wanting to design the system to be > fast from the beginning. That is almost certainly better > than to design something, realize it is way to slow, and then > hack on it forever. However, having this working in the > short term and rewriting it for the long term doesn't upset > me too much -- I just want it working, and it will certainly > be good enough for a large range of applications (even it > it isn't large enough or good enough for Terry). It won't be better than a user space call conversion scheduler for the vast majority of threaded applications. I've been through the benchmarks on the code that was posted to -current and on the similar SVR4 N:N kernel threading model and the SVR4 M:N, M>N "lets starve all the user space threads from getting quantum" Solaris 2.3 and UnixWare 2.x model. It's not a question of "not good enough", it's a question of "if you aren't intending on following the implementation through to completion, there's no reason to bother starting down the road at all". I guess what I'm trying to point out is that there is a crisis of commitment; merely having kernel threads won't make the code go faster. SMP scalability is not merely the ability to block between threads waiting on each others resources in the Big Giant Lock(tm) in the kernel. You can block on the User Space Call Conversion Scheduler(tm) instead, and achieve exactly the same (lack of) effect. If you are truly interested in pursing SMP scalability via kernel threads, the way to do it is to take the Dyson vfork() code, run it on your own machine, and work up from there. Insisting that FreeBSD commit from a non-SMP scalable call conversion model in favor of a non-SMP scalable context switch and cache busting kernel threading model is not the way to go. To get anywhere with that argument, you are going to have to be able to beat the user space threads with your kernel space threads on a uniprocessor system, and show imporvement, or at least no degradation, on an SMP system from using the user space scheduler. Kernel threads context switches are *not* lighter weight than process context switches. The cost is about equal, unless you have some way of assuring CPU <-> threads group affinity. Even then, you are talking about starving other processes in favor of the thread group unless you are very, very careful implementing your code. The only place this won't be true is a rigged benchmark on an otherwise idle machine, such that your benchmark process never has to compete with any other process, and so the page table is never pushed out to make way for the page table for "init" or "syncd" or "nfsiod", etc., etc.. This is not a trivial problem to solve, and hitting the currently limping-but-functional user space threads over the head with a shovel and dragging the body away to make room for a different and limping-but-even-less-functional new body won't cut it. Ugh. I need to drag my IEEE SMP and parallel processing literature out of my back bedroom when I get home tonight, I guess. Then I'll be able to quote you chapter and verse. 8-(. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 15 19:38:01 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id TAA13828 for freebsd-smp-outgoing; Tue, 15 Dec 1998 19:38:01 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from fast.cs.utah.edu (fast.cs.utah.edu [155.99.212.1]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id TAA13822 for ; Tue, 15 Dec 1998 19:37:54 -0800 (PST) (envelope-from vanmaren@fast.cs.utah.edu) Received: (from vanmaren@localhost) by fast.cs.utah.edu (8.9.1/8.9.1) id UAA08980; Tue, 15 Dec 1998 20:37:46 -0700 (MST) Date: Tue, 15 Dec 1998 20:37:46 -0700 (MST) From: Kevin Van Maren Message-Id: <199812160337.UAA08980@fast.cs.utah.edu> To: tlambert@primenet.com Subject: Re: Pthreads and SMP Cc: smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > The problem is write faults on pages that are in the page tables > of both processors because there is only one page table for all > of the threads in a single process. I think we are in agreement... If a thread on processor A causes the mappings to be made less restrictive, then (worst case) processor B gets a page fault because its TLB was out-of-date. No big deal, the kernel just notices and has the thread continue. A TLB- shootdown is only necessary when you take away access permission to ensure that the no-longer-allowed access is disabled on the other processor. > Kernel threads context switches are *not* lighter weight than process > context switches. The cost is about equal, unless you have some way > of assuring CPU <-> threads group affinity. Even then, you are talking Switching between two threads in the same process is faster, all things being equal, because you don't have to flush the TLB on the x86 (reload CR3). But you are right -- just selecting a random thread probably won't necessarially pick one from the same process. > These mechanisms already exist, as has been pointed out countless > times on this list. They just aren't packaged up with their glue code > in a nice "pthread_create" routine some place, because doing that I guess I should look at the patches. I've written pthreads code before; hacking pthread_create from the mechanisms shouldn't be too bad. But I'm trying very hard to finish my thesis now :-) > without further kernel support would result in abysmal performance, > and would, in fact, be counter-productive. No, if 99% of the work to get it "running" is already there, I don't think it is a waste. It will work until it is replaced with something better. My point is that kernel-threads are better than the user-only thread because they allow multiple threads in the same process to execute at the same time. Whether or not this is always a win is irrelevant -- the programmer should be able to have two threads scheduled concurrently. I also think that saying "we have kernel threads" is a great advertisement, even if they DON'T help performance tremendously. Being able to do pthread_create() and peg both processors with a single process sounds good to me. Of course, the global kernel lock is going to start causing serious performance problems before much longer. I've already seen cases where the system time approached 50% on my dual Pentium II. Kevin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 15 22:52:48 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id WAA01753 for freebsd-smp-outgoing; Tue, 15 Dec 1998 22:52:48 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from post.mail.demon.net (post-12.mail.demon.net [194.217.242.41]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id WAA01743; Tue, 15 Dec 1998 22:52:44 -0800 (PST) (envelope-from james@westongold.com) Received: from [158.152.96.124] (helo=wgp01.wgold.demon.co.uk) by post.mail.demon.net with esmtp (Exim 2.054 #1) id 0zqA1q-00047h-00; Wed, 16 Dec 1998 06:01:46 +0000 Received: by WGP01 with Internet Mail Service (5.5.1960.3) id ; Wed, 16 Dec 1998 06:11:25 -0000 Message-ID: <32BABEF63EAED111B2C5204C4F4F5020183C@WGP01> From: James Mansion To: Terry Lambert , James Mansion Cc: mal@algonet.se, alk@pobox.com, peter@netplex.com.au, gpalmer@FreeBSD.ORG, marcelk@stack.nl, smp@FreeBSD.ORG Subject: RE: Pthreads and SMP Date: Wed, 16 Dec 1998 06:11:20 -0000 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.1960.3) Content-Type: text/plain; charset="iso-8859-1" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > -----Original Message----- > From: Terry Lambert [mailto:tlambert@primenet.com] > ... > > So, in what way is an application built from multiple processes with > > shared state in a shared memory segment going to be better than an > > application with multiple kernel threads running in a largely shared > > address space? > > I don't understand. Why do you need to share state? > > If you need to share state, then why do you need threads instead of > a select-driven finite state automaton? > > I guess when all you have is a threads implementation, everything > looks like a thread... > > For a shared context server, the viability of the server is based > on how *little* context you actually have to contend for between > processors. Look, I don't give a fig for irrelevant petty tasks like file serving. If your example was of a modern DBMS kernel I'd have more time for it. I'm sorry that you can't understand that actually the whole point of these silly computer things is to process data rather than just move it. My personal interest is twofold: - I want to compute while IO is under way - I want to speed computations (especially monte-carlo calcs) that can be split, on SMP boxes In the former case, even a good async IO implementation will not help without all the open and sync functions being available through aio - and they aren't. And select/poll isn't even close, given the behaviour with 'local' file fds. In the latter, I have a non-trivial data structure (possibly large) that will be read during computation. Its stupid to build it from scratch in multiple processes, and threads serve well. In practice, DBMS systems are also compute bound, and also have significant shared state. Its not just investment banking or engineering weenies who have to deal with this. > If all the address space is shared, then an invalidation of an > area in the address space by a thread on CPU #5 makes updates to > the other CPU's, even though the threads running on those CPU's > may never actually use the data. > > It's more an artifact of having a common page table between all > of the processors, such that the same invalidations affect all > of the processors instead of one processor. And you have evidence that, in the case where primary cache lines are not being invalidated in a contended region, that this has a major effect? You don't think that the eveidence of scalability on real, cheap Intel-based kit with SMP DBMS shows that the effect need not be a major headache, and that useful gains can be made, even with the 8th CPU and even with NT? > Yes, it's a problem when you share context between threads, no > matter how you do it. The moral to the story is "don't do that". > If a problem is capable of being parallelized, then it makes sense > to try and compute it in parallel; if it isn't, then no amount of > adding PC's together is going to get you a virtual supercomputer. That's right. And quite a lot can be parallelised, actually. Its just been so ugly and painful with SysV shared memory that no-one bothered except for DBMS vendors. > > The problem space mappable using SMP is not trivially small, but > it's a hell of a lot smaller than the problem space that can be > mapped with 32 times the computations per clock cycle on a linear > uniprocessor. And this observation is practical given Intel's attack on 4- and 8-way volume servers? I don't think so. > It's my personal experience that people use threads because they > don't know how to program effectively, and using threads efficiently > requires a lot of saftey harness code in the OS for these people to > actually gain the benefit they think they will gain. Its MY personal experience that you should get a life. The OS is only there to make our lives easier, so we can do our real jobs, no? James To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 16 07:00:41 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id HAA24741 for freebsd-smp-outgoing; Wed, 16 Dec 1998 07:00:41 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from server.noc.demon.net (server.noc.demon.net [193.195.224.4]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id HAA24733 for ; Wed, 16 Dec 1998 07:00:35 -0800 (PST) (envelope-from fanf@demon.net) Received: by server.noc.demon.net; id PAA05982; Wed, 16 Dec 1998 15:00:23 GMT Received: from fanf.noc.demon.net(195.11.55.83) by inside.noc.demon.net via smap (3.2) id xmac05931; Wed, 16 Dec 98 15:00:09 GMT Received: from fanf by fanf.noc.demon.net with local (Exim 1.73 #2) id 0zqIVn-0000OX-00; Wed, 16 Dec 1998 15:05:15 +0000 To: smp@FreeBSD.ORG From: Tony Finch Subject: Re: Pthreads and SMP Newsgroups: chiark.mail.freebsd.smp In-Reply-To: Organization: Deliberate Obfuscation To Amuse Tony References: <199812150732.AAA12768@usr06.primenet.com> Message-Id: Date: Wed, 16 Dec 1998 15:05:15 +0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org "Robert S. Sciuk" wrote: > >The select() mechanism and asynch io is also a viable, albeit conceptually >more difficult to implement for the average programmer. The main problem with select() with disk IO is that you are still limited by seek times. Squid gets around this problem in the latest version by using threading, but this is no use if the threading libray is userland-only :-( Tony. -- 1421421**f.a.n.finch dot@dotat.at fanf@demon.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 16 08:13:01 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id IAA03724 for freebsd-smp-outgoing; Wed, 16 Dec 1998 08:13:01 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from bright.fx.genx.net (bright.fx.genx.net [206.64.4.154]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id IAA03719 for ; Wed, 16 Dec 1998 08:12:57 -0800 (PST) (envelope-from bright@hotjobs.com) Received: from localhost (bright@localhost) by bright.fx.genx.net (8.9.1/8.9.1) with ESMTP id LAA06250; Wed, 16 Dec 1998 11:17:30 -0500 (EST) (envelope-from bright@hotjobs.com) X-Authentication-Warning: bright.fx.genx.net: bright owned process doing -bs Date: Wed, 16 Dec 1998 11:17:30 -0500 (EST) From: Alfred Perlstein X-Sender: bright@bright.fx.genx.net To: Kevin Van Maren cc: tlambert@primenet.com, smp@FreeBSD.ORG Subject: Re: Pthreads and SMP In-Reply-To: <199812160337.UAA08980@fast.cs.utah.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, 15 Dec 1998, Kevin Van Maren wrote: > > without further kernel support would result in abysmal performance, > > and would, in fact, be counter-productive. > > No, if 99% of the work to get it "running" is already there, I don't > think it is a waste. It will work until it is replaced with something > better. My point is that kernel-threads are better than the > user-only thread because they allow multiple threads in the same > process to execute at the same time. Whether or not this is > always a win is irrelevant -- the programmer should be able to > have two threads scheduled concurrently. > > I also think that saying "we have kernel threads" is a great > advertisement, even if they DON'T help performance tremendously. > Being able to do pthread_create() and peg both processors with > a single process sounds good to me. Just a simple question, don't you _gain_ performance when you schedule threads on multiple processors if the threads do not manipulate data that is close together? Basically you can either split one processor's cache between several threads and hope the exibit locality, or split on several processors and hope for spatial accesses? I'm just saying that even though it may be better served by seperate processes, threaded programs that don't have much contention could really win when threads are spread across CPUs. I think that Terry has pointed out several problems with the scheduler including CPU affinity which makes this argument invalid, however if that aspect is fixed it may really be/eventually become a gain. (please excuse the mangling of the thread 'buzzwords' as i don't have any referances handy :) ) -Alfred > > Of course, the global kernel lock is going to start causing serious > performance problems before much longer. I've already seen cases > where the system time approached 50% on my dual Pentium II. > > Kevin > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 16 14:49:53 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id OAA24752 for freebsd-smp-outgoing; Wed, 16 Dec 1998 14:49:53 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id OAA24738 for ; Wed, 16 Dec 1998 14:49:49 -0800 (PST) (envelope-from tlambert@usr09.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.8.8/8.8.8) id PAA11160; Wed, 16 Dec 1998 15:49:37 -0700 (MST) Received: from usr09.primenet.com(206.165.6.209) via SMTP by smtp02.primenet.com, id smtpd011031; Wed Dec 16 15:49:26 1998 Received: (from tlambert@localhost) by usr09.primenet.com (8.8.5/8.8.5) id PAA04662; Wed, 16 Dec 1998 15:49:15 -0700 (MST) From: Terry Lambert Message-Id: <199812162249.PAA04662@usr09.primenet.com> Subject: Re: Pthreads and SMP To: dot@dotat.at (Tony Finch) Date: Wed, 16 Dec 1998 22:49:14 +0000 (GMT) Cc: smp@FreeBSD.ORG In-Reply-To: from "Tony Finch" at Dec 16, 98 03:05:15 pm X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > >The select() mechanism and asynch io is also a viable, albeit conceptually > >more difficult to implement for the average programmer. > > The main problem with select() with disk IO is that you are still > limited by seek times. Squid gets around this problem in the latest > version by using threading, but this is no use if the threading libray > is userland-only :-( Explicitly ask for read ahead on things you will need in the future: madvise( ..., ..., MADV_SEQUENTIAL); madvise( ..., ..., MADV_WILLNEED); Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 16 16:11:27 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id QAA07216 for freebsd-smp-outgoing; Wed, 16 Dec 1998 16:11:27 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from smtp01.primenet.com (smtp01.primenet.com [206.165.6.131]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id QAA07207; Wed, 16 Dec 1998 16:11:23 -0800 (PST) (envelope-from tlambert@usr09.primenet.com) Received: (from daemon@localhost) by smtp01.primenet.com (8.8.8/8.8.8) id RAA29657; Wed, 16 Dec 1998 17:11:04 -0700 (MST) Received: from usr09.primenet.com(206.165.6.209) via SMTP by smtp01.primenet.com, id smtpd029374; Wed Dec 16 17:11:00 1998 Received: (from tlambert@localhost) by usr09.primenet.com (8.8.5/8.8.5) id RAA07668; Wed, 16 Dec 1998 17:10:12 -0700 (MST) From: Terry Lambert Message-Id: <199812170010.RAA07668@usr09.primenet.com> Subject: Re: Pthreads and SMP To: james@westongold.com (James Mansion) Date: Thu, 17 Dec 1998 00:10:12 +0000 (GMT) Cc: tlambert@primenet.com, james@westongold.com, mal@algonet.se, alk@pobox.com, peter@netplex.com.au, gpalmer@FreeBSD.ORG, marcelk@stack.nl, smp@FreeBSD.ORG In-Reply-To: <32BABEF63EAED111B2C5204C4F4F5020183C@WGP01> from "James Mansion" at Dec 16, 98 06:11:20 am X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > For a shared context server, the viability of the server is based > > on how *little* context you actually have to contend for between > > processors. > > Look, I don't give a fig for irrelevant petty tasks like file serving. Don't think about it like this. The train of thought is too limiting. Think of CPU cycles as if they were a resource. > I'm sorry that you can't understand that actually the whole point of > these silly computer things is to process data rather than just move > it. > > My personal interest is twofold: > - I want to compute while IO is under way Use prefetch notification (via madvise) and use John Birrell's AIO patches to -current. > - I want to speed computations (especially monte-carlo calcs) that > can be split, on SMP boxes I have some familiarity with this problem, on SMP SPARC hardware, since I have worked on SMP capable software for simulating relativistically invariant particle collisions via monte-carlo, with the resulting collisions being constrained to "reasonableness" by the physics being modelled (a soloution of 12 Feynman-Dyson diagrams). >From personal experience with the LBL code for relativistically invariant collisions on a dual P90 FreeBSD box, starting on June 19th, 1995 using the October 27th, 1995 snapshot for FreeBSD, when I first got a FreeBSD box to run SMP using Jack Vogel's initial code, I can tell you that SMP scalability is very poor even to today using kernel threads vs. seperate processes. > In the former case, even a good async IO implementation will not > help without all the open and sync functions being available through > aio - and they aren't. And select/poll isn't even close, given the > behaviour with 'local' file fds. Right. This is my main complaint with async I/O, especially as has been codified by POSIX. My personal preference? Use async call gates, and return context references for calls which block. Divorce the kernel stack from the user space process entirely. A putative kernel threads implementaiton won't help you much in this case, either, since it only buys you call contexts on which to block. Big deal, you get to be descheduled in favor of "syncd" or "swapper" or some other process that doesn't have your page table in its heart of hearts. You might as well be running in a seperate process, since the context switch overhead is going to be the same, and, at least with a seperate process, you don't have to pay for TLB shootdown. I guess the I/O assumptions you are making about interleave are assuming you are using SCSI, right? ...since IDE devices will serialize the I/O at the IDE interfaces, whereas the SCSI devices can interleave I/O requests to the device using a tagged command queue. Otherwise, you might as well be running in a select loop. > In the latter, I have a non-trivial data structure (possibly large) > that will be read during computation. Its stupid to build it from > scratch in multiple processes, and threads serve well. So does a single mmap'ed file or a SYSV SHM segment loaded by the initial process. What you seem to be complaining about here is page table setup and teardown, since once the pages are in core, the pages are shared (FreeBSD has a unified VM and buffer cache, and the vnode *directly* backs the object, unlike SVR4 where you would pay a buffer-cache to VM copy overhead for each process. > In practice, DBMS systems are also compute bound, and also have > significant shared state. Its not just investment banking or > engineering weenies who have to deal with this. Actually, this is an architecture issue that has so far been ignored when people go to implement their code. I think the correct soloution for that *particular* problem would be to export a dependency registration mechanism from the Soft Updates code such that user transactions could be marked to occur in dependency order as well. This would be a general win, and ensure that the DBMS I/O was *not* I/O bottlenecked to the disk. This type of thing is why the DBMS weenies all get so hot and bothered about log structured and journalling FS technology. The "compute binding" of DBMS technology is in the transaction latency, not in actual computational latency. > > It's more an artifact of having a common page table between all > > of the processors, such that the same invalidations affect all > > of the processors instead of one processor. > > And you have evidence that, in the case where primary cache lines > are not being invalidated in a contended region, that this has a > major effect? Yes. Empirical evidence from physics simulations, which is what I assume you are planning on using the thing for, given your reference to monte-carlo. Specifically, the fact that IPI's are generated at L2 cache speed instead of the actual processor speed. > You don't think that the eveidence of scalability on real, cheap > Intel-based kit with SMP DBMS shows that the effect need not be > a major headache, and that useful gains can be made, even with > the 8th CPU and even with NT? No, I don't. Most SMP DBMS scalability has been implemented using what SVR4 calls "scheduling classes" -- and they've addressed these issues archtecturally. For example, if you were going to do a monte-carlo run of the type discussed above, you would have to seperate the work into discrete components. DBMS's do this intrinsically, implementing "work to do" engines that sperate context from connection, and that's why they actually can get performance wins on SMP: they are designed and tuned for the limitations of the SMP on which they are expected to run. > > The problem space mappable using SMP is not trivially small, but > > it's a hell of a lot smaller than the problem space that can be > > mapped with 32 times the computations per clock cycle on a linear > > uniprocessor. > > And this observation is practical given Intel's attack on 4- and > 8-way volume servers? I don't think so. The only good Intel SMP boes are the ones that come out of their server products group, traditionally, and Intel doesn't sell them to people, they only kick them out as reference implemetnations. It's a strange world. The idea behind the referenced "attack" is the same idea behind the claim that you can add a bunch of PC's together and get a supercomputer. > > It's my personal experience that people use threads because they > > don't know how to program effectively, and using threads efficiently > > requires a lot of saftey harness code in the OS for these people to > > actually gain the benefit they think they will gain. > > Its MY personal experience that you should get a life. The OS is > only there to make our lives easier, so we can do our real jobs, > no? Yes, it is. And you can probably complain loudly enough that people will implement the API's you want for kernel threading in -current, instead of making you type a list search pattern for the -current list archives to find the kernel threading glue code on your own. But there is going to be no advantage to you for having done this, and FreeBSD is going to look bad as a result, since it'll be nothing more than a facade. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 16 23:46:08 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id XAA19411 for freebsd-smp-outgoing; Wed, 16 Dec 1998 23:46:08 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from ns3.redbacknetworks.com (mail3.redbacknetworks.com [155.53.200.100]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id XAA19405 for ; Wed, 16 Dec 1998 23:46:06 -0800 (PST) (envelope-from sam@redbacknetworks.com) Received: from phred.redbacknetworks.com (phred.redbacknetworks.com [155.53.144.35]) by ns3.redbacknetworks.com (8.8.8/8.8.8) with ESMTP id XAA12990 for ; Wed, 16 Dec 1998 23:46:00 -0800 (PST) Received: (from sam@localhost) by phred.redbacknetworks.com (8.8.5/8.8.5) id XAA21791; Wed, 16 Dec 1998 23:45:14 -0800 (PST) Date: Wed, 16 Dec 1998 23:45:14 -0800 (PST) Message-Id: <199812170745.XAA21791@phred.redbacknetworks.com> From: Sam Pigg To: smp@FreeBSD.ORG Subject: Re: tyan S1836DLUAN >512Mb problems Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org >> Trying to use 1 gig of memory on a tyan S1836DLUAN (dual 450 PII's) >> running 3.0-RELEASE, keep having problems with the system freezing up, >> rebooting, compiles failing due to signal 11's etc. If I decrease the >> memory to 512 megs (I'm using those weird micron 256meg simms btw) >> everything works fine. Tyan claims this motherboard can support up to >> '1GB SPD SDRAM or SDRAM+ECC in addition to the new 100MHz >> SDRAMs' >> >> This motherboard uses the 440GX chipset, but another system with >> a tyan S1832DL (440BX chipset) suffers from the same symptoms when >> >512 megs of ram are installed. > >The problem is not with the motherboard. Most of the mother boards that >support SMP PII's have the same problem. I was unable to locate where in >the configs the problem comes from though. I was testing dual PII 400 >when they were first avalible from Compaq with 1.0 gig of 256meg dimms. >They are recognized fine by the bios but FreeBSD won't work with it. I >tested NT, Novell, and Sco all of which work although no where near as >well and the system with 512meg and SMP FreeBSD. > >Rob. I've upgraded the board bios, changed to a new memory vendor, etc etc. Still have exactly the same symptoms, and from what rob said it would seem this problem is specific to freebsd, and not any particular motherboard. Is anyone actually running a freebsd machine with more than 512 megs of memory? Anyone have a clue at all as to why freebsd can't support a gig of ram? windblows seems to have no problem (which doesnt help me much) Thanks, Sam To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 17 01:27:49 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id BAA29352 for freebsd-smp-outgoing; Thu, 17 Dec 1998 01:27:49 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from csvax.cs.caltech.edu (csvax.cs.caltech.edu [131.215.131.131]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id BAA29347 for ; Thu, 17 Dec 1998 01:27:46 -0800 (PST) (envelope-from mika@varese.cs.caltech.edu) Received: from varese.cs.caltech.edu (varese.cs.caltech.edu [131.215.78.28]) by csvax.cs.caltech.edu (8.9.1/8.9.1) with ESMTP id BAA27022; Thu, 17 Dec 1998 01:27:37 -0800 (PST) Received: from localhost.cs.caltech.edu (localhost.cs.caltech.edu [127.0.0.1]) by varese.cs.caltech.edu (8.8.7/8.7.3) with SMTP id BAA17946; Thu, 17 Dec 1998 01:27:33 -0800 (PST) Message-Id: <199812170927.BAA17946@varese.cs.caltech.edu> X-Authentication-Warning: varese.cs.caltech.edu: localhost.cs.caltech.edu [127.0.0.1] didn't use HELO protocol To: Sam Pigg Cc: smp@FreeBSD.ORG Subject: Re: tyan S1836DLUAN >512Mb problems In-reply-to: Your message of "Wed, 16 Dec 1998 23:45:14 PST." <199812170745.XAA21791@phred.redbacknetworks.com> Date: Thu, 17 Dec 1998 01:27:33 -0800 From: Mika Nystroem Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Got a couple of old Intel quad-capable but dual Alder machines here with a gigabyte of interleaved FPM memory in them, running 3.0. (We had one running with quad processors for a while but another group needed their processors.) Mika Sam Pigg writes: > >>> Trying to use 1 gig of memory on a tyan S1836DLUAN (dual 450 PII's) >>> running 3.0-RELEASE, keep having problems with the system freezing up, >>> rebooting, compiles failing due to signal 11's etc. If I decrease the >>> memory to 512 megs (I'm using those weird micron 256meg simms btw) >>> everything works fine. Tyan claims this motherboard can support up to >>> '1GB SPD SDRAM or SDRAM+ECC in addition to the new 100MHz >>> SDRAMs' >>> >>> This motherboard uses the 440GX chipset, but another system with >>> a tyan S1832DL (440BX chipset) suffers from the same symptoms when >>> >512 megs of ram are installed. >> >>The problem is not with the motherboard. Most of the mother boards that >>support SMP PII's have the same problem. I was unable to locate where in >>the configs the problem comes from though. I was testing dual PII 400 >>when they were first avalible from Compaq with 1.0 gig of 256meg dimms. >>They are recognized fine by the bios but FreeBSD won't work with it. I >>tested NT, Novell, and Sco all of which work although no where near as >>well and the system with 512meg and SMP FreeBSD. >> >>Rob. > > >I've upgraded the board bios, changed to a new memory vendor, etc etc. >Still have exactly the same symptoms, and from what rob said it would >seem this problem is specific to freebsd, and not any particular motherboard. > >Is anyone actually running a freebsd machine with more than 512 megs of >memory? Anyone have a clue at all as to why freebsd can't support >a gig of ram? windblows seems to have no problem (which doesnt help me much) > >Thanks, >Sam > >To Unsubscribe: send mail to majordomo@FreeBSD.org >with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 17 04:35:08 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id EAA19548 for freebsd-smp-outgoing; Thu, 17 Dec 1998 04:35:08 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from server.noc.demon.net (server.noc.demon.net [193.195.224.4]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id EAA19499 for ; Thu, 17 Dec 1998 04:34:56 -0800 (PST) (envelope-from fanf@demon.net) Received: by server.noc.demon.net; id MAA09635; Thu, 17 Dec 1998 12:34:46 GMT Received: from fanf.noc.demon.net(195.11.55.83) by inside.noc.demon.net via smap (3.2) id xma009631; Thu, 17 Dec 98 12:34:36 GMT Received: from fanf by fanf.noc.demon.net with local (Exim 1.73 #2) id 0zqcdX-0000zI-00; Thu, 17 Dec 1998 12:34:35 +0000 To: smp@FreeBSD.ORG From: Tony Finch Subject: Re: Pthreads and SMP Newsgroups: chiark.mail.freebsd.smp In-Reply-To: <199812162249.PAA04662@usr09.primenet.com> Organization: Deliberate Obfuscation To Amuse Tony References: Message-Id: Date: Thu, 17 Dec 1998 12:34:35 +0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Terry Lambert wrote: >> >> The main problem with select() with disk IO is that you are still >> limited by seek times. Squid gets around this problem in the latest >> version by using threading, but this is no use if the threading libray >> is userland-only :-( > >Explicitly ask for read ahead on things you will need in the future: > > madvise( ..., ..., MADV_SEQUENTIAL); > madvise( ..., ..., MADV_WILLNEED); Is there any way of doing a non-blocking open() of a disk file? O_NONBLOCK (aka O_NDELAY elsewhere) is only advertised as doing the right thing in this situation on Linux as far as I have seen. Tony. -- ocmcocmcocmk**f.a.n.finch fanf@demon.net dot@dotat.at To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 17 10:54:04 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id KAA00788 for freebsd-smp-outgoing; Thu, 17 Dec 1998 10:54:04 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from ns3.redbacknetworks.com (mail3.redbacknetworks.com [155.53.200.100]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id KAA00742 for ; Thu, 17 Dec 1998 10:54:01 -0800 (PST) (envelope-from sam@redbacknetworks.com) Received: from phred.redbacknetworks.com (phred.redbacknetworks.com [155.53.144.35]) by ns3.redbacknetworks.com (8.8.8/8.8.8) with ESMTP id KAA17554 for ; Thu, 17 Dec 1998 10:53:54 -0800 (PST) Received: (from sam@localhost) by phred.redbacknetworks.com (8.8.5/8.8.5) id KAA22205; Thu, 17 Dec 1998 10:53:08 -0800 (PST) Date: Thu, 17 Dec 1998 10:53:08 -0800 (PST) Message-Id: <199812171853.KAA22205@phred.redbacknetworks.com> From: Sam Pigg To: freebsd-smp@FreeBSD.ORG Subject: Re: tyan S1836DLUAN >512Mb problems Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org more data I probably should've included- the new memory I'm using is from the tyan approved/tested vendor list for this motherboard (dense-pac 256 Meg ECC SDRAM dimms) The loading of the system I'm doing uses thousands of processes, which I can run more of with 512M than 1Gig. ie it fires off the processes slowly until it runs out of physical ram and then moves through swap when using 512M, with 1Gig, it freezes when it gets down to 350-255 megs free of physical ram. Tried juggling the new dimms around to different dimm slots, to no avail. I have a difficult time believing its a memory bus loading issue as this is ram thats actually been tested at tyan. Also have logs of vmstat -i -s -m and 'top' taken every second while loading up the processes before the system freeze, that show no apparent resource starvation going on. Any ideas? Thanks, -Sam To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 17 14:01:54 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id OAA24261 for freebsd-smp-outgoing; Thu, 17 Dec 1998 14:01:54 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from ns3.redbacknetworks.com (mail3.redbacknetworks.com [155.53.200.100]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id OAA24247 for ; Thu, 17 Dec 1998 14:01:52 -0800 (PST) (envelope-from sam@rback.com) Received: from phred.redbacknetworks.com (phred.redbacknetworks.com [155.53.144.35]) by ns3.redbacknetworks.com (8.8.8/8.8.8) with ESMTP id OAA18655; Thu, 17 Dec 1998 14:01:42 -0800 (PST) Received: from rback.com (localhost [127.0.0.1]) by phred.redbacknetworks.com (8.8.5/8.8.5) with ESMTP id OAA22371; Thu, 17 Dec 1998 14:00:57 -0800 (PST) Message-ID: <36797F19.8DA010A8@rback.com> Date: Thu, 17 Dec 1998 14:00:57 -0800 From: Sam X-Mailer: Mozilla 4.03 [en] (X11; I; SunOS 5.5.1 sun4u) MIME-Version: 1.0 To: Tor.Egge@fast.no CC: smp@FreeBSD.ORG Subject: Re: tyan S1836DLUAN >512Mb problems References: <199812171853.KAA22205@phred.redbacknetworks.com> <199812171943.UAA17794@midten.fast.no> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Added those options to the config, and changed pmap.h to double the SMP size of NKPDE as well as the LOAD_BASE in the Makefile. System would reboot immediately upon loading the kernel. Is there something else I would need to do for this? Thanks, -Sam Tor.Egge@fast.no wrote: > > > The loading of the system I'm doing uses thousands of processes, which > > I can run more of with 512M than 1Gig. ie it fires off the processes > > slowly until it runs out of physical ram and then moves through swap > > when using 512M, with 1Gig, it freezes when it gets down to > > 350-255 megs free of physical ram. Tried juggling the new dimms > > around to different dimm slots, to no avail. > > In the vmstat -m output, you have memory statistics by type. If the > usage for a type hits the limit, malloc will block, and the result > might be a deadlock/freeze. > > I suggest adding > > options "VM_KMEM_SIZE=(128*1024*1024)" > options "VM_KMEM_SIZE_MAX=(128*1024*1024)" > > to the kernel config file, and doubling the kernel kva range > (from 252 MB to 508 MB, see PR#1880 for a pre-SMP patch). > > - Tor Egge To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 17 17:12:19 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id RAA17452 for freebsd-smp-outgoing; Thu, 17 Dec 1998 17:12:19 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id RAA17446 for ; Thu, 17 Dec 1998 17:12:16 -0800 (PST) (envelope-from tlambert@usr08.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.8.8/8.8.8) id SAA18972; Thu, 17 Dec 1998 18:12:04 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp02.primenet.com, id smtpd018915; Thu Dec 17 18:11:59 1998 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id SAA19059; Thu, 17 Dec 1998 18:11:54 -0700 (MST) From: Terry Lambert Message-Id: <199812180111.SAA19059@usr08.primenet.com> Subject: Re: Pthreads and SMP To: dot@dotat.at (Tony Finch) Date: Fri, 18 Dec 1998 01:11:53 +0000 (GMT) Cc: smp@FreeBSD.ORG In-Reply-To: from "Tony Finch" at Dec 17, 98 12:34:35 pm X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > >> The main problem with select() with disk IO is that you are still > >> limited by seek times. Squid gets around this problem in the latest > >> version by using threading, but this is no use if the threading libray > >> is userland-only :-( > > > >Explicitly ask for read ahead on things you will need in the future: > > > > madvise( ..., ..., MADV_SEQUENTIAL); > > madvise( ..., ..., MADV_WILLNEED); > > Is there any way of doing a non-blocking open() of a disk file? > O_NONBLOCK (aka O_NDELAY elsewhere) is only advertised as doing the > right thing in this situation on Linux as far as I have seen. What's a "non-blocking open of a disk file"? In other words, other than FIFO's or special devices, what would you expect to block? Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 17 18:03:20 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id SAA23909 for freebsd-smp-outgoing; Thu, 17 Dec 1998 18:03:20 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from dingo.cdrom.com (dingo.cdrom.com [204.216.28.145]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id SAA23904 for ; Thu, 17 Dec 1998 18:03:19 -0800 (PST) (envelope-from mike@dingo.cdrom.com) Received: from dingo.cdrom.com (localhost.cdrom.com [127.0.0.1]) by dingo.cdrom.com (8.9.1/8.8.8) with ESMTP id SAA00482; Thu, 17 Dec 1998 18:01:00 -0800 (PST) (envelope-from mike@dingo.cdrom.com) Message-Id: <199812180201.SAA00482@dingo.cdrom.com> X-Mailer: exmh version 2.0.2 2/24/98 To: Terry Lambert cc: dot@dotat.at (Tony Finch), smp@FreeBSD.ORG Subject: Re: Pthreads and SMP In-reply-to: Your message of "Fri, 18 Dec 1998 01:11:53 GMT." <199812180111.SAA19059@usr08.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 17 Dec 1998 18:00:59 -0800 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > >> The main problem with select() with disk IO is that you are still > > >> limited by seek times. Squid gets around this problem in the latest > > >> version by using threading, but this is no use if the threading libray > > >> is userland-only :-( > > > > > >Explicitly ask for read ahead on things you will need in the future: > > > > > > madvise( ..., ..., MADV_SEQUENTIAL); > > > madvise( ..., ..., MADV_WILLNEED); > > > > Is there any way of doing a non-blocking open() of a disk file? > > O_NONBLOCK (aka O_NDELAY elsewhere) is only advertised as doing the > > right thing in this situation on Linux as far as I have seen. > > What's a "non-blocking open of a disk file"? > > In other words, other than FIFO's or special devices, what would > you expect to block? Disk or network activity. But for an open to truly "not block", you have to change the semantics slightly (especially, you have to require a close on a not-really-open file). -- \\ Sometimes you're ahead, \\ Mike Smith \\ sometimes you're behind. \\ mike@smith.net.au \\ The race is long, and in the \\ msmith@freebsd.org \\ end it's only with yourself. \\ msmith@cdrom.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 17 18:26:54 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id SAA26900 for freebsd-smp-outgoing; Thu, 17 Dec 1998 18:26:54 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from sv01.cet.co.jp (sv01.cet.co.jp [210.171.56.2]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id SAA26889 for ; Thu, 17 Dec 1998 18:26:52 -0800 (PST) (envelope-from michaelh@cet.co.jp) Received: from localhost (michaelh@localhost) by sv01.cet.co.jp (8.8.8/8.8.8) with SMTP id CAA22142; Fri, 18 Dec 1998 02:26:35 GMT (envelope-from michaelh@cet.co.jp) Date: Fri, 18 Dec 1998 11:26:35 +0900 (JST) From: Michael Hancock To: Kevin Van Maren cc: tlambert@primenet.com, smp@FreeBSD.ORG Subject: Re: Pthreads and SMP In-Reply-To: <199812160337.UAA08980@fast.cs.utah.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, 15 Dec 1998, Kevin Van Maren wrote: > > Kernel threads context switches are *not* lighter weight than process > > context switches. The cost is about equal, unless you have some way > > of assuring CPU <-> threads group affinity. Even then, you are talking > > Switching between two threads in the same process is faster, > all things being equal, because you don't have to flush the > TLB on the x86 (reload CR3). But you are right -- just selecting > a random thread probably won't necessarially pick one from the > same process. How much more would it cost to have the scheduler favor a thread in a process that had another thread block while a lot of quantum was left, over an async call gate implementation? > Of course, the global kernel lock is going to start causing serious > performance problems before much longer. I've already seen cases > where the system time approached 50% on my dual Pentium II. I think we should go to subsystem coarse-grained locks as a first step and allow different SMP approaches within each subsystem. Regards, Mike Hancock To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 17 19:45:07 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id TAA07481 for freebsd-smp-outgoing; Thu, 17 Dec 1998 19:45:07 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id TAA07476 for ; Thu, 17 Dec 1998 19:45:06 -0800 (PST) (envelope-from tlambert@usr06.primenet.com) Received: (from daemon@localhost) by smtp04.primenet.com (8.8.8/8.8.8) id UAA24711; Thu, 17 Dec 1998 20:44:48 -0700 (MST) Received: from usr06.primenet.com(206.165.6.206) via SMTP by smtp04.primenet.com, id smtpd024592; Thu Dec 17 20:44:43 1998 Received: (from tlambert@localhost) by usr06.primenet.com (8.8.5/8.8.5) id UAA20700; Thu, 17 Dec 1998 20:44:30 -0700 (MST) From: Terry Lambert Message-Id: <199812180344.UAA20700@usr06.primenet.com> Subject: Re: Pthreads and SMP To: michaelh@cet.co.jp (Michael Hancock) Date: Fri, 18 Dec 1998 03:44:29 +0000 (GMT) Cc: vanmaren@fast.cs.utah.edu, tlambert@primenet.com, smp@FreeBSD.ORG In-Reply-To: from "Michael Hancock" at Dec 18, 98 11:26:35 am X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Switching between two threads in the same process is faster, > > all things being equal, because you don't have to flush the > > TLB on the x86 (reload CR3). But you are right -- just selecting > > a random thread probably won't necessarially pick one from the > > same process. > > How much more would it cost to have the scheduler favor a thread in a > process that had another thread block while a lot of quantum was left, > over an async call gate implementation? The quantum clock is reset on a context switch, so it would cost you the ability to run any other program on the system until such time as all threads in that program block or ran to the point of quantum expiration, at which point an involuntary context switch would occur. Of course, given an involuntary context switch occurring, there's no reason why the scheduler wouldn't potentially pick a different thread in the same process, so you could get CPU starvation of all other programs that way, too. > > Of course, the global kernel lock is going to start causing serious > > performance problems before much longer. I've already seen cases > > where the system time approached 50% on my dual Pentium II. > > I think we should go to subsystem coarse-grained locks as a first step and > allow different SMP approaches within each subsystem. No. We need to address the problem (kernel reentrancy), not the symptom (waiting on locks for a non-reentrant kernel). Kernel threads are an inelegant kludge if they are being used to address the kernel reentrancy issue; that issue is much better addressed by maintaining a pool of contexts within the kernel on which user space processes make system call requests and the kernel executes this on behalf of the user process. The problem of SMP scaling is totally seperate. If you consider CPU cycles as a consumable resource, it's a pretty simple problem to deal with allocation of CPU cycles to code that wants to run. First, you run all of the kernel code that's ready to run, then you run all of the user space code that wants to run. CPU cycles can be spread wherever they are needed, without regard to whether it's kernel or user space code that's being run. Typically, you would expect that, barring system work, the CPU's are running in user space code. Better to not address SMP scalability through explicit thread to CPU mapping, except as necessary to minimize bus contention between the processors (affinity, etc.). At this point, the whole value to "how many threads in a process" has more to do with the concurrency that a process can achieve, than in the method by which concurrency is achieved. The idea of "kernel threads allocated to a process" is not a policy or specific implementation concern, but rather a measure of how aggressively a given process will compete for quantum against other processes in the system. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 18 02:13:56 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id CAA19633 for freebsd-smp-outgoing; Fri, 18 Dec 1998 02:13:56 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from sv01.cet.co.jp (sv01.cet.co.jp [210.171.56.2]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id CAA19620 for ; Fri, 18 Dec 1998 02:13:50 -0800 (PST) (envelope-from michaelh@cet.co.jp) Received: from localhost (michaelh@localhost) by sv01.cet.co.jp (8.8.8/8.8.8) with SMTP id KAA24284; Fri, 18 Dec 1998 10:13:30 GMT (envelope-from michaelh@cet.co.jp) Date: Fri, 18 Dec 1998 19:13:30 +0900 (JST) From: Michael Hancock To: Terry Lambert cc: vanmaren@fast.cs.utah.edu, smp@FreeBSD.ORG Subject: Re: Pthreads and SMP In-Reply-To: <199812180344.UAA20700@usr06.primenet.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > How much more would it cost to have the scheduler favor a thread in a > > process that had another thread block while a lot of quantum was left, > > over an async call gate implementation? > > The quantum clock is reset on a context switch, so it would cost > you the ability to run any other program on the system until such > time as all threads in that program block or ran to the point of > quantum expiration, at which point an involuntary context switch > would occur. > > Of course, given an involuntary context switch occurring, there's > no reason why the scheduler wouldn't potentially pick a different > thread in the same process, so you could get CPU starvation of all > other programs that way, too. I think the scheduler in BSD favors I/O bound processes anyway so the process should get another quantum soon. Long running CPU bound processes get lower priorities but if they don't get enough CPU then the priority is boosted. The Async call gate does sound interesting for process contention threads. It seems intuitive that a process should be allowed to use its threads to use up as much of its quantum as possible. But instead of an async call gate you should probably be thinking in terms of "scheduler activations" where the kernel informs the Pthreads library that it's about to block a thread. The library can immediately schedule a new Pthreads thread. This is what Digital UNIX does. Solaris 2.5 would just block and programmers had to allocate extra kernel entities (LWPs?) to maintain process concurrency. > > > Of course, the global kernel lock is going to start causing serious > > > performance problems before much longer. I've already seen cases > > > where the system time approached 50% on my dual Pentium II. > > > > I think we should go to subsystem coarse-grained locks as a first step and > > allow different SMP approaches within each subsystem. > > No. We need to address the problem (kernel reentrancy), not the > symptom (waiting on locks for a non-reentrant kernel). I think that can be worked on gradually per-subsystem so why not get the immediate gratification of a coarse grained system while you're working towards that goal. Making the kernel reentrant is a lot of work and presumably the guy working on the network code will be different from the guy working on the fs code. > Kernel threads are an inelegant kludge if they are being used to > address the kernel reentrancy issue; that issue is much better > addressed by maintaining a pool of contexts within the kernel on > which user space processes make system call requests and the > kernel executes this on behalf of the user process. > > The problem of SMP scaling is totally seperate. If you consider > CPU cycles as a consumable resource, it's a pretty simple problem > to deal with allocation of CPU cycles to code that wants to run. > > First, you run all of the kernel code that's ready to run, then > you run all of the user space code that wants to run. CPU cycles > can be spread wherever they are needed, without regard to whether > it's kernel or user space code that's being run. Typically, you > would expect that, barring system work, the CPU's are running in > user space code. > > Better to not address SMP scalability through explicit thread > to CPU mapping, except as necessary to minimize bus contention > between the processors (affinity, etc.). > > At this point, the whole value to "how many threads in a process" > has more to do with the concurrency that a process can achieve, > than in the method by which concurrency is achieved. The idea > of "kernel threads allocated to a process" is not a policy or > specific implementation concern, but rather a measure of how > aggressively a given process will compete for quantum against > other processes in the system. With Pthreads you have the options: pthread_attr_setscope( ..., PTHREAD_SCOPE_PROCESS); or pthread_attr_setscope( ..., PTHREAD_SCOPE_SYSTEM); You have to implement one, but it'd be nice to have both. Process contention is good for most apps, but if you need more predictability then you need system contention where the threads will compete with threads in other processes. Regards, Mike Hancock To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 18 03:58:58 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id DAA28488 for freebsd-smp-outgoing; Fri, 18 Dec 1998 03:58:58 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from server.noc.demon.net (server.noc.demon.net [193.195.224.4]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id DAA28480 for ; Fri, 18 Dec 1998 03:58:56 -0800 (PST) (envelope-from fanf@demon.net) Received: by server.noc.demon.net; id LAA16736; Fri, 18 Dec 1998 11:58:46 GMT Received: from fanf.noc.demon.net(195.11.55.83) by inside.noc.demon.net via smap (3.2) id xma016728; Fri, 18 Dec 98 11:58:32 GMT Received: from fanf by fanf.noc.demon.net with local (Exim 1.73 #2) id 0zqyYB-0001yv-00; Fri, 18 Dec 1998 11:58:31 +0000 To: smp@FreeBSD.ORG From: Tony Finch Subject: Re: Pthreads and SMP Newsgroups: chiark.mail.freebsd.smp In-Reply-To: <199812180111.SAA19059@usr08.primenet.com> Organization: Deliberate Obfuscation To Amuse Tony References: Message-Id: Date: Fri, 18 Dec 1998 11:58:31 +0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Terry Lambert wrote: > >In other words, other than FIFO's or special devices, what would >you expect to block? Seeks are the main problem: single-threaded Squid is limited to about 100 URLs per second because it spends all its time in biowait. Tony. -- f.a.n.finch**??id6890 dot@dotat.at fanf@demon.net To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 18 22:23:34 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id WAA01367 for freebsd-smp-outgoing; Fri, 18 Dec 1998 22:23:34 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from thelab.hub.org ([142.177.186.246]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id WAA01362 for ; Fri, 18 Dec 1998 22:23:33 -0800 (PST) (envelope-from scrappy@hub.org) Received: from localhost (scrappy@localhost) by thelab.hub.org (8.9.1/8.9.1) with ESMTP id CAA06608 for ; Sat, 19 Dec 1998 02:23:31 -0400 (AST) (envelope-from scrappy@hub.org) X-Authentication-Warning: thelab.hub.org: scrappy owned process doing -bs Date: Sat, 19 Dec 1998 02:23:30 -0400 (AST) From: The Hermit Hacker To: freebsd-smp@FreeBSD.ORG Subject: Quad motherboards? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I'm looking around for a Quad server, and can't seem to find anything but over-priced Compaq computers... :( Anyone know whether Asus or similar produce one, that I just can't find references to? Intel? Thanks... Marc G. Fournier Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 18 23:54:08 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id XAA08821 for freebsd-smp-outgoing; Fri, 18 Dec 1998 23:54:08 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from bokonon.logiclink.com (BOKONON.LOGICLINK.COM [206.215.203.38]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id XAA08806 for ; Fri, 18 Dec 1998 23:54:05 -0800 (PST) (envelope-from irving@bokonon.logiclink.com) Received: from localhost (irving@localhost) by bokonon.logiclink.com (8.9.1/8.9.1) with ESMTP id CAA37081; Sat, 19 Dec 1998 02:54:02 -0500 (EST) (envelope-from irving@bokonon.logiclink.com) Date: Sat, 19 Dec 1998 02:54:01 -0500 (EST) From: Irving Popovetsky To: The Hermit Hacker cc: freebsd-smp@FreeBSD.ORG Subject: Re: Quad motherboards? In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Since the Quad Processor phenomenon is now coming back in style only with the release of the PII Xeon chips, it seems like most of the major motherboard manufacturers have just now managed to release dual processor motherboards, but are slow with the quads. Intel, of course, does make one, its called the SC450NX .... you can oooh at it: http://developer.intel.com/design/servers/sc450nx/index.htm As far as people who are actually selling it, I havent seen all that many, but I noticed a company in Placentia, CA seems to be peddling it. They must have hired some crazy photographers: http://www.areasys.com/products/motherboards/SC450NXphoto.htm Frankly, it kind of frightens me. Knowing intel's track record of very hot processors, and seeing 4 Xeons all bunched together like that makes me think of some kind of hellfire-deathtrap. Computers have come full-circle, I guess. Again we have come to a day when you have to point a big fan at your computer. Hope this helps, -Irving Popovetsky Systems Administrator, Logic Solutions Inc. On Sat, 19 Dec 1998, The Hermit Hacker wrote: > Date: Sat, 19 Dec 1998 02:23:30 -0400 (AST) > From: The Hermit Hacker > To: freebsd-smp@FreeBSD.ORG > Subject: Quad motherboards? > > > I'm looking around for a Quad server, and can't seem to find anything but > over-priced Compaq computers... :( > > Anyone know whether Asus or similar produce one, that I just can't find > references to? Intel? > > Thanks... > > Marc G. Fournier > Systems Administrator @ hub.org > primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Dec 19 10:14:31 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id KAA01454 for freebsd-smp-outgoing; Sat, 19 Dec 1998 10:14:31 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from Ilsa.StevesCafe.com (Ilsa.StevesCafe.com [205.168.119.129]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id KAA01449 for ; Sat, 19 Dec 1998 10:14:29 -0800 (PST) (envelope-from fbsd@Ilsa.StevesCafe.com) Received: from Ilsa.StevesCafe.com (localhost.StevesCafe.com [127.0.0.1]) by Ilsa.StevesCafe.com (8.8.8/8.8.5) with ESMTP id LAA05886; Sat, 19 Dec 1998 11:25:02 -0700 (MST) Message-Id: <199812191825.LAA05886@Ilsa.StevesCafe.com> X-Mailer: exmh version 2.0.2 2/24/98 From: Steve Passe To: The Hermit Hacker cc: freebsd-smp@FreeBSD.ORG Subject: Re: Quad motherboards? In-reply-to: Your message of "Sat, 19 Dec 1998 02:23:30 -0400." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 19 Dec 1998 11:25:01 -0700 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, > > I'm looking around for a Quad server, and can't seem to find anything but > over-priced Compaq computers... :( > > Anyone know whether Asus or similar produce one, that I just can't find > references to? Intel? http://www.atipa.com/cgi-bin/systembuilder?sc450nx -- Steve Passe | powered by smp@csn.net | Symmetric MultiProcessor FreeBSD To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Dec 19 11:10:43 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id LAA06760 for freebsd-smp-outgoing; Sat, 19 Dec 1998 11:10:43 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from www1.asacomputers.com ([204.153.176.244]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id LAA06755 for ; Sat, 19 Dec 1998 11:10:42 -0800 (PST) (envelope-from kedar@asacomputers.com) Received: from kedar.asacomputers.com (kedar.asacomputers.com [204.153.176.86]) by www1.asacomputers.com (8.8.8/8.8.8) with SMTP id LAA12038; Sat, 19 Dec 1998 11:10:50 -0800 (PST) (envelope-from kedar@asacomputers.com) Message-Id: <2.2.32.19981219190629.013d1494@gw1> X-Sender: rajadnya@gw1 X-Mailer: Windows Eudora Pro Version 2.2 (32) Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Date: Sat, 19 Dec 1998 11:06:29 -0800 To: The Hermit Hacker From: Kedar Rajadnya Subject: Re: Quad motherboards? Cc: freebsd-smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org >I'm looking around for a Quad server, and can't seem to find anything but >over-priced Compaq computers... :( >Anyone know whether Asus or similar produce one, that I just can't find >references to? Intel? Hello, Intel makes 2, one which you will find less cost-prohibitive: the SCS450NX. AMI makes one, the MegaPLEX. It does cost more than the Intel box, though. Holds more memory, more PCI slots, etc. They are not offered as motherboards alone. They are a whole kit with enclosure, 80-pin SCSI backplane, memory boards, etc. Take care, Kedar Rajadnya. ASA Computers, Inc. TEL: (408)232-5999 ext201 FAX:(408)232-5959 ********************************************* To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Dec 19 21:28:12 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id VAA05754 for freebsd-smp-outgoing; Sat, 19 Dec 1998 21:28:12 -0800 (PST) (envelope-from owner-freebsd-smp@FreeBSD.ORG) Received: from dingo.cdrom.com (castles151.castles.com [208.214.165.151]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id VAA05747 for ; Sat, 19 Dec 1998 21:28:08 -0800 (PST) (envelope-from mike@dingo.cdrom.com) Received: from dingo.cdrom.com (LOCALHOST [127.0.0.1]) by dingo.cdrom.com (8.9.1/8.8.8) with ESMTP id VAA41483; Sat, 19 Dec 1998 21:25:49 -0800 (PST) (envelope-from mike@dingo.cdrom.com) Message-Id: <199812200525.VAA41483@dingo.cdrom.com> X-Mailer: exmh version 2.0.2 2/24/98 To: Irving Popovetsky cc: The Hermit Hacker , freebsd-smp@FreeBSD.ORG Subject: Re: Quad motherboards? In-reply-to: Your message of "Sat, 19 Dec 1998 02:54:01 EST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 19 Dec 1998 21:25:49 -0800 From: Mike Smith Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Since the Quad Processor phenomenon is now coming back in style only with > the release of the PII Xeon chips, it seems like most of the major > motherboard manufacturers have just now managed to release dual processor > motherboards, but are slow with the quads. > > Intel, of course, does make one, its called the SC450NX .... you can > oooh at it: > http://developer.intel.com/design/servers/sc450nx/index.htm They have another, larger boardset. > Frankly, it kind of frightens me. Knowing intel's track record of very > hot processors, and seeing 4 Xeons all bunched together like that makes me > think of some kind of hellfire-deathtrap. Computers have come > full-circle, I guess. Again we have come to a day when you have to point a big > fan at your computer. If you buy the chassis as well as the boardset from Intel, you get a pile of nice ducting and lots of fans with it. The AD450NX system here has six of them, with onboard temperature sensing feeding back into a speed controller. (There are 3 more fans cooling the I/O board on the other side.) -- \\ Sometimes you're ahead, \\ Mike Smith \\ sometimes you're behind. \\ mike@smith.net.au \\ The race is long, and in the \\ msmith@freebsd.org \\ end it's only with yourself. \\ msmith@cdrom.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message