From owner-freebsd-smp Sun Dec 15 01:10:39 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id BAA06365 for smp-outgoing; Sun, 15 Dec 1996 01:10:39 -0800 (PST) Received: from tfs.com (tfs.com [140.145.250.1]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id BAA06348; Sun, 15 Dec 1996 01:10:33 -0800 (PST) Received: from critter.tfs.com by tfs.com (smail3.1.28.1) with SMTP id m0vZCa9-0003w0C; Sun, 15 Dec 96 01:10 PST Received: from critter.tfs.com (localhost.phk.dk [127.0.0.1]) by critter.tfs.com (8.8.2/8.8.2) with ESMTP id KAA09094; Sun, 15 Dec 1996 10:13:28 +0100 (MET) To: Peter Wemm cc: dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com Subject: Re: some questions concerning TLB shootdowns in FreeBSD In-reply-to: Your message of "Sun, 15 Dec 1996 09:21:55 +0800." <199612150121.JAA12763@spinner.DIALix.COM> Date: Sun, 15 Dec 1996 10:13:27 +0100 Message-ID: <9092.850641207@critter.tfs.com> From: Poul-Henning Kamp Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk In message <199612150121.JAA12763@spinner.DIALix.COM>, Peter Wemm writes: >However, the shared address space code that I was working on in >-current (for kernel assisted threading in the smp kernel) means >that a single vmspace/pmap/etc can be shared among multiple processes >and this changes the above picture since two cpu's can be using >the user mode parts of the same page tables at once, one in executing >in user mode, one in the kernel. But we could still have a per-cpu flags: "I'm not in a shared address-space" Ie, this would only be set if the CPU was in userland in a non-threaded process. -- Poul-Henning Kamp | phk@FreeBSD.ORG FreeBSD Core-team. http://www.freebsd.org/~phk | phk@login.dknet.dk Private mailbox. whois: [PHK] | phk@tfs.com TRW Financial Systems, Inc. Power and ignorance is a disgusting cocktail. From owner-freebsd-smp Sun Dec 15 02:05:40 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id CAA13610 for smp-outgoing; Sun, 15 Dec 1996 02:05:40 -0800 (PST) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id CAA13517; Sun, 15 Dec 1996 02:05:04 -0800 (PST) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.8.4/8.8.4) with ESMTP id SAA14741; Sun, 15 Dec 1996 18:03:18 +0800 (WST) Message-Id: <199612151003.SAA14741@spinner.DIALix.COM> To: Poul-Henning Kamp cc: dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com Subject: Re: some questions concerning TLB shootdowns in FreeBSD In-reply-to: Your message of "Sun, 15 Dec 1996 10:13:27 +0100." <9092.850641207@critter.tfs.com> Date: Sun, 15 Dec 1996 18:03:16 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Poul-Henning Kamp wrote: > In message <199612150121.JAA12763@spinner.DIALix.COM>, Peter Wemm writes: > > >However, the shared address space code that I was working on in > >-current (for kernel assisted threading in the smp kernel) means > >that a single vmspace/pmap/etc can be shared among multiple processes > >and this changes the above picture since two cpu's can be using > >the user mode parts of the same page tables at once, one in executing > >in user mode, one in the kernel. > > But we could still have a per-cpu flags: > "I'm not in a shared address-space" > > Ie, this would only be set if the CPU was in userland in a non-threaded > process. eg, something like: if (is_userland && curproc->p_vmspace->vm_refcnt > 1) send_tlb_invalidate(); Cheers, -Peter From owner-freebsd-smp Sun Dec 15 02:14:25 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id CAA14248 for smp-outgoing; Sun, 15 Dec 1996 02:14:25 -0800 (PST) Received: from tfs.com (tfs.com [140.145.250.1]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id CAA14243; Sun, 15 Dec 1996 02:14:18 -0800 (PST) Received: from critter.tfs.com by tfs.com (smail3.1.28.1) with SMTP id m0vZDZq-0003vuC; Sun, 15 Dec 96 02:13 PST Received: from critter.tfs.com (localhost.phk.dk [127.0.0.1]) by critter.tfs.com (8.8.2/8.8.2) with ESMTP id LAA09321; Sun, 15 Dec 1996 11:17:14 +0100 (MET) To: Peter Wemm cc: dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com Subject: Re: some questions concerning TLB shootdowns in FreeBSD In-reply-to: Your message of "Sun, 15 Dec 1996 18:03:16 +0800." <199612151003.SAA14741@spinner.DIALix.COM> Date: Sun, 15 Dec 1996 11:17:14 +0100 Message-ID: <9319.850645034@critter.tfs.com> From: Poul-Henning Kamp Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk In message <199612151003.SAA14741@spinner.DIALix.COM>, Peter Wemm writes: >Poul-Henning Kamp wrote: >> In message <199612150121.JAA12763@spinner.DIALix.COM>, Peter Wemm writes: >> >> >However, the shared address space code that I was working on in >> >-current (for kernel assisted threading in the smp kernel) means >> >that a single vmspace/pmap/etc can be shared among multiple processes >> >and this changes the above picture since two cpu's can be using >> >the user mode parts of the same page tables at once, one in executing >> >in user mode, one in the kernel. >> >> But we could still have a per-cpu flags: >> "I'm not in a shared address-space" >> >> Ie, this would only be set if the CPU was in userland in a non-threaded >> process. > >eg, something like: >if (is_userland && curproc->p_vmspace->vm_refcnt > 1) > send_tlb_invalidate(); Well more like: for (i=0;ip_vmspace->vm_refcnt > 1) send_tlb_invalidate(i); -- Poul-Henning Kamp | phk@FreeBSD.ORG FreeBSD Core-team. http://www.freebsd.org/~phk | phk@login.dknet.dk Private mailbox. whois: [PHK] | phk@tfs.com TRW Financial Systems, Inc. Power and ignorance is a disgusting cocktail. From owner-freebsd-smp Sun Dec 15 02:50:00 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id CAA17093 for smp-outgoing; Sun, 15 Dec 1996 02:50:00 -0800 (PST) Received: from spinner.DIALix.COM (root@spinner.DIALix.COM [192.203.228.67]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id CAA17068; Sun, 15 Dec 1996 02:49:48 -0800 (PST) Received: from spinner.DIALix.COM (peter@localhost.DIALix.oz.au [127.0.0.1]) by spinner.DIALix.COM (8.8.4/8.8.4) with ESMTP id SAA19672; Sun, 15 Dec 1996 18:49:29 +0800 (WST) Message-Id: <199612151049.SAA19672@spinner.DIALix.COM> To: Poul-Henning Kamp cc: dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com Subject: Re: some questions concerning TLB shootdowns in FreeBSD In-reply-to: Your message of "Sun, 15 Dec 1996 11:17:14 +0100." <9319.850645034@critter.tfs.com> Date: Sun, 15 Dec 1996 18:49:28 +0800 From: Peter Wemm Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Poul-Henning Kamp wrote: > In message <199612151003.SAA14741@spinner.DIALix.COM>, Peter Wemm writes: > >Poul-Henning Kamp wrote: > >> In message <199612150121.JAA12763@spinner.DIALix.COM>, Peter Wemm writes: > >> > >> >However, the shared address space code that I was working on in > >> >-current (for kernel assisted threading in the smp kernel) means > >> >that a single vmspace/pmap/etc can be shared among multiple processes > >> >and this changes the above picture since two cpu's can be using > >> >the user mode parts of the same page tables at once, one in executing > >> >in user mode, one in the kernel. > >> > >> But we could still have a per-cpu flags: > >> "I'm not in a shared address-space" > >> > >> Ie, this would only be set if the CPU was in userland in a non-threaded > >> process. > > > >eg, something like: > >if (is_userland && curproc->p_vmspace->vm_refcnt > 1) > > send_tlb_invalidate(); > > Well more like: > > for (i=0;i if (is_userland(i) && curproc[i]->p_vmspace->vm_refcnt > 1) > send_tlb_invalidate(i); I'm not sure that this is needed.. After all, we're deciding whether we have to flush the TLB of remote processors when fiddling with the user level components of the local current process. If the local process is not shared, we can ignore what the other cpu's are doing. Anyway, it's probably more correct to say: if (!is_userland || curproc....vm_refcnt > 1) smp_invltlb(); since as a general rule we'd always want to arrange a tlb flush for kernel accesses. I simplified the comment originally because for the kernel accesses, we can simply set a bit to cause the other cpu's to invalidate their tlb on entry to the kernel when they go for the mutex lock. But, for accesses to non-shared process user spaces that are not curproc on the local cpu, we must do something like this: for (i = 0; i < NCPU; i++) { if (i != cpnumber() && cpu_present[i] && SMPcurproc[i] == p) flush_bits |= 1 << i; } if (flush_bits) freeze_cpu_mask(flush_bits); fiddle_non_current_process_PTE() if (flush_bits) wake_cpu_mask(flush_bits); etc. I can tell that John is going to have a heart attack over this.. :-) Cheers, -Peter From owner-freebsd-smp Sun Dec 15 03:03:53 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id DAA18396 for smp-outgoing; Sun, 15 Dec 1996 03:03:53 -0800 (PST) Received: from hda.hda.com (ip27-max1-fitch.ziplink.net [199.232.245.27]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id DAA18391 for ; Sun, 15 Dec 1996 03:03:48 -0800 (PST) Received: (from dufault@localhost) by hda.hda.com (8.6.12/8.6.12) id FAA17314; Sun, 15 Dec 1996 05:38:19 -0500 From: Peter Dufault Message-Id: <199612151038.FAA17314@hda.hda.com> Subject: Re: SMP Status In-Reply-To: from Kory Hamzeh at "Dec 14, 96 05:36:07 pm" To: kory@avatar.com (Kory Hamzeh) Date: Sun, 15 Dec 1996 05:38:19 -0500 (EST) Cc: freebsd-smp@freebsd.org X-Mailer: ELM [version 2.4ME+ PL25 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Is there anywhere on the freebsd web site where I can get a status of the > SMP project? I putting togther a fairly high end Pentium Pro machine > right now and I would like to purchase a motherboard that would be > compatible with the freebsd SMP support. Look in ~fsmp on FreeBSD.org. The home automation page is down there too. Hopefully with all the new web manpower a link can show up from the home page. -- Peter Dufault (dufault@hda.com) Real-Time Machine Control and Simulation HD Associates, Inc. Voice: 508 433 6936 From owner-freebsd-smp Sun Dec 15 08:28:55 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id IAA13535 for smp-outgoing; Sun, 15 Dec 1996 08:28:55 -0800 (PST) Received: from dyson.iquest.net (dyson.iquest.net [198.70.144.127]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id IAA13524; Sun, 15 Dec 1996 08:28:49 -0800 (PST) Received: (from root@localhost) by dyson.iquest.net (8.8.2/8.6.9) id LAA05010; Sun, 15 Dec 1996 11:28:31 -0500 (EST) From: "John S. Dyson" Message-Id: <199612151628.LAA05010@dyson.iquest.net> Subject: Re: some questions concerning TLB shootdowns in FreeBSD To: phk@critter.tfs.com (Poul-Henning Kamp) Date: Sun, 15 Dec 1996 11:28:31 -0500 (EST) Cc: peter@spinner.dialix.com, dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com In-Reply-To: <9319.850645034@critter.tfs.com> from "Poul-Henning Kamp" at Dec 15, 96 11:17:14 am X-Mailer: ELM [version 2.4 PL24 ME8] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > In message <199612151003.SAA14741@spinner.DIALix.COM>, Peter Wemm writes: > >Poul-Henning Kamp wrote: > >> In message <199612150121.JAA12763@spinner.DIALix.COM>, Peter Wemm writes: > >> > >> >However, the shared address space code that I was working on in > >> >-current (for kernel assisted threading in the smp kernel) means > >> >that a single vmspace/pmap/etc can be shared among multiple processes > >> >and this changes the above picture since two cpu's can be using > >> >the user mode parts of the same page tables at once, one in executing > >> >in user mode, one in the kernel. > >> > >> But we could still have a per-cpu flags: > >> "I'm not in a shared address-space" > >> > >> Ie, this would only be set if the CPU was in userland in a non-threaded > >> process. > > > >eg, something like: > >if (is_userland && curproc->p_vmspace->vm_refcnt > 1) > > send_tlb_invalidate(); > > Well more like: > > for (i=0;i if (is_userland(i) && curproc[i]->p_vmspace->vm_refcnt > 1) > send_tlb_invalidate(i); > This won't work because processes seldom have the entire address space shared (vm_refcnt.) I am sure that when we get true kernel multithreading that will not be true though. In order to test if a section of an address space is shared, you have to do something like this (and this can take LOTS of time.) (I might have levels of indirection off here, I am also not taking into account submaps -- which complicate the code further, by entailing recursively calling the map/object traversal again -- but recursion is a major no-no in the kernel, as we have found.) for(i=0;ip_vmspace->vm_map->header.next; while (tmpe != &map->header) { if ((tmpe->is_sub_map == 0) && (tmpe->is_a_map == 0)) { tobj = tmpe->object.vm_object; while(tobj) { if (tobj->ref_count > 1) { send_tlb_invalidate(i); goto nextcpu; } tobj = tobj->backing_object; } } else { /* * here on a new map -- need to recurse or * somesuch here -- yuck!!! */ } tmpe = tmpe->next; } nextcpu: } John From owner-freebsd-smp Sun Dec 15 08:54:40 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id IAA14893 for smp-outgoing; Sun, 15 Dec 1996 08:54:40 -0800 (PST) Received: from dyson.iquest.net (dyson.iquest.net [198.70.144.127]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id IAA14881; Sun, 15 Dec 1996 08:54:33 -0800 (PST) Received: (from root@localhost) by dyson.iquest.net (8.8.2/8.6.9) id LAA05078; Sun, 15 Dec 1996 11:54:22 -0500 (EST) From: "John S. Dyson" Message-Id: <199612151654.LAA05078@dyson.iquest.net> Subject: Re: some questions concerning TLB shootdowns in FreeBSD To: toor@dyson.iquest.net (John S. Dyson) Date: Sun, 15 Dec 1996 11:54:22 -0500 (EST) Cc: phk@critter.tfs.com, peter@spinner.dialix.com, dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com In-Reply-To: <199612151628.LAA05010@dyson.iquest.net> from "John S. Dyson" at Dec 15, 96 11:28:31 am X-Mailer: ELM [version 2.4 PL24 ME8] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > > > In message <199612151003.SAA14741@spinner.DIALix.COM>, Peter Wemm writes: > > >Poul-Henning Kamp wrote: > > >> In message <199612150121.JAA12763@spinner.DIALix.COM>, Peter Wemm writes: > > >> > > >> >However, the shared address space code that I was working on in > > >> >-current (for kernel assisted threading in the smp kernel) means > > >> >that a single vmspace/pmap/etc can be shared among multiple processes > > >> >and this changes the above picture since two cpu's can be using > > >> >the user mode parts of the same page tables at once, one in executing > > >> >in user mode, one in the kernel. > > >> ^^^^^^^^^^^^^^^ > > >> But we could still have a per-cpu flags: > > >> "I'm not in a shared address-space" > > >> > > >> Ie, this would only be set if the CPU was in userland in a non-threaded > > >> process. > > > > > >eg, something like: > > >if (is_userland && curproc->p_vmspace->vm_refcnt > 1) > > > send_tlb_invalidate(); > > > > Well more like: > > > > for (i=0;i > if (is_userland(i) && curproc[i]->p_vmspace->vm_refcnt > 1) > > send_tlb_invalidate(i); > > > This won't work because processes seldom have the entire address space > shared (vm_refcnt.) I am sure that when we get true kernel multithreading > that will not be true though. In order to test if a section of an > address space is shared, you have to do something like this (and > this can take LOTS of time.) (I might have levels of indirection > off here, I am also not taking into account submaps -- which > complicate the code further, by entailing recursively calling > the map/object traversal again -- but recursion is a major > no-no in the kernel, as we have found.) > Note that I do see that you were talking about shared address spaces, but address spaces are already partially shared. To do the thing completely requires traversing alot of the VM data structures. I would suggest that a coarser grained scheme for pmap_update (invtlb) be considered in the case of SMP. Also (Peter's ?) suggestion that we have individual alternate page table's (and temporary mapping pages) for each CPU's has merit. It is likely that large numbers of TLB flushes could be eliminated if the above were implemented. Since global TLB flushes are going to be fairly expensive, let's minimize them -- but scanning the VM data structures is going to be expensive no matter how we do it. Note that I have put individual page invalidates into pmap -- we need to usually remove those in the SMP code. (There are some special mapping pages where we should probably continue doing the page invalidates -- but those should also be per-cpu.) John From owner-freebsd-smp Sun Dec 15 12:56:34 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id MAA29285 for smp-outgoing; Sun, 15 Dec 1996 12:56:34 -0800 (PST) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id MAA29253; Sun, 15 Dec 1996 12:56:23 -0800 (PST) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id NAA23823; Sun, 15 Dec 1996 13:32:03 -0700 From: Terry Lambert Message-Id: <199612152032.NAA23823@phaeton.artisoft.com> Subject: Re: some questions concerning TLB shootdowns in FreeBSD To: toor@dyson.iquest.net (John S. Dyson) Date: Sun, 15 Dec 1996 13:32:03 -0700 (MST) Cc: toor@dyson.iquest.net, phk@critter.tfs.com, peter@spinner.dialix.com, dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com In-Reply-To: <199612151654.LAA05078@dyson.iquest.net> from "John S. Dyson" at Dec 15, 96 11:54:22 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > This won't work because processes seldom have the entire address space > > shared (vm_refcnt.) I am sure that when we get true kernel multithreading > > that will not be true though. In order to test if a section of an > > address space is shared, you have to do something like this (and > > this can take LOTS of time.) (I might have levels of indirection > > off here, I am also not taking into account submaps -- which > > complicate the code further, by entailing recursively calling > > the map/object traversal again -- but recursion is a major > > no-no in the kernel, as we have found.) > > > > Note that I do see that you were talking about shared address spaces, > but address spaces are already partially shared. To do the thing completely > requires traversing alot of the VM data structures. I would suggest > that a coarser grained scheme for pmap_update (invtlb) be considered in > the case of SMP. Also (Peter's ?) suggestion that we have individual > alternate page table's (and temporary mapping pages) for each CPU's > has merit. > > It is likely that large numbers of TLB flushes could be eliminated > if the above were implemented. Since global TLB flushes are going to > be fairly expensive, let's minimize them -- but scanning the VM > data structures is going to be expensive no matter how we do it. > > Note that I have put individual page invalidates into pmap -- we > need to usually remove those in the SMP code. (There are some > special mapping pages where we should probably continue doing > the page invalidates -- but those should also be per-cpu.) Some potential optimizations: 1) This only applys to written pages not marked copy-on-write; read-only pages and pages that will be copied on write (like those in your note about "address spaces are already shared") 2) Flushing can be "lazy" in most cases. That is, the page could be marked invalid for a particular CPU, and only flushed if that CPU needs to use it. For a generic first time implementation, a single unsigned long with CPU invalidity bits could be used (first time because it places a 32 processor limit, which I feel is an unacceptable limitation -- I want to run on Connection Machines some day) as an addition to the page attributes. For the most part, it is important to realize that this is a negative validity indicator. This dictates who has to do the work: the CPU that wants to access the page. The higher the process CPU affinity, the less this will happen. 3) For processes with shared address space, the common data area for all CPU's should be grown. Yes, I realize this means a seperate virtual address space for CPU private, CPU shared, and per process user space addressing. This is less of a burden than you might thing, if you divorce the kernel stack from the idea of process and place it squarely on the head of kernel threads. For a blocking call, a per CPU thread pool can be used as a context container. This would require another bitmap on sleep events so that the CPU's affected can be notified without blocking everyone. 4) One obvious consequence of a per SPU thread pool approach (which I also think is acceptable) is that the wakeup must be processed on the CPU on which it went to sleep. Theoretically, this should mean very little, since the CPU returning a kernel thread is not necessarily being bound to the process. Practically, it probably means that rebinding the CPU for a process can only occur on a blocking system call entry (by choosing which processor to acquire the kernel thread to handle the blocking call) or on involuntary context switch. This is acceptable, since a process making a blocking call or for which the CPU has been involuntarily relinquished will not feel the calculation overhead of the decision; it will be buried in the latency before it is next scheduled to run. For an involuntary context switch, this takes the form of picking which CPU's run queue to insert the process on. Regards, Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Sun Dec 15 16:59:32 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id QAA24420 for smp-outgoing; Sun, 15 Dec 1996 16:59:32 -0800 (PST) Received: from dyson.iquest.net (dyson.iquest.net [198.70.144.127]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id QAA24404; Sun, 15 Dec 1996 16:59:24 -0800 (PST) Received: (from root@localhost) by dyson.iquest.net (8.8.2/8.6.9) id TAA05793; Sun, 15 Dec 1996 19:58:05 -0500 (EST) From: "John S. Dyson" Message-Id: <199612160058.TAA05793@dyson.iquest.net> Subject: Re: some questions concerning TLB shootdowns in FreeBSD To: terry@lambert.org (Terry Lambert) Date: Sun, 15 Dec 1996 19:58:05 -0500 (EST) Cc: toor@dyson.iquest.net, phk@critter.tfs.com, peter@spinner.dialix.com, dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com In-Reply-To: <199612152032.NAA23823@phaeton.artisoft.com> from "Terry Lambert" at Dec 15, 96 01:32:03 pm X-Mailer: ELM [version 2.4 PL24 ME8] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Terry, I am NOT discounting your suggestions -- but I am bringing up challenges associated with your suggestions. > > Some potential optimizations: > > 1) This only applys to written pages not marked copy-on-write; > read-only pages and pages that will be copied on write (like > those in your note about "address spaces are already shared") > You still need to be able to globally invalidate pages that are mapped read-only. (e.g. paging out (where things can happen anytime, or object reclaimation -- that of course doesn't happen unless an object is done with.)) > 2) Flushing can be "lazy" in most cases. That is, the page could > be marked invalid for a particular CPU, and only flushed if > that CPU needs to use it. For a generic first time implementation, > a single unsigned long with CPU invalidity bits could be used > (first time because it places a 32 processor limit, which I > feel is an unacceptable limitation -- I want to run on Connection > Machines some day) as an addition to the page attributes. For > the most part, it is important to realize that this is a > negative validity indicator. This dictates who has to do > the work: the CPU that wants to access the page. The higher > the process CPU affinity, the less this will happen. > There is no special indication of a TLB entry being updated in a processor from the page tables. So, once there is a page table entry created, we have no indication when the processor grabs it (I seem to remember that there are ways for coercing P6's and perhaps P5's into doing it though.) The only way that I would try to do it is to get information from Intel saying that the method is "blessed." It could break things for other (non-Intel) CPU's though. John From owner-freebsd-smp Sun Dec 15 18:47:26 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id SAA02376 for smp-outgoing; Sun, 15 Dec 1996 18:47:26 -0800 (PST) Received: from parkplace.cet.co.jp (parkplace.cet.co.jp [202.32.64.1]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id SAA02371 for ; Sun, 15 Dec 1996 18:47:23 -0800 (PST) Received: from localhost (michaelh@localhost) by parkplace.cet.co.jp (8.8.3/CET-v2.1) with SMTP id CAA24713 for ; Mon, 16 Dec 1996 02:47:20 GMT Date: Mon, 16 Dec 1996 11:47:20 +0900 (JST) From: Michael Hancock To: smp@freebsd.org Subject: General SMP Design In-Reply-To: <199612160058.TAA05793@dyson.iquest.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk This paper was brought up on hackers and I think it has some good guidelines for SMP design. I am not advocating Non-blocking Synchronization yet, because I still don't know enough about it, but much of what is written is good for SMP in general. http://www-dsg.stanford.edu/papers/non-blocking-osdi/index.html We have many examples of type-stable memory management in our code. i.e. a vnode is always an instance of a vnode, it can be a on a free list. It doesn't change it's type. The zone allocator is a good thing. The other interesting things aside from NBS included CMDS, contention minimizing data structures. They advocate per-processor pools which many agree is good for SMP performance. I've noticed a lot of headers regarding TLB shootdowns recently. The text "Unix Internals: The New Frontiers" by Uresh Vahalia has some algorithms. Regards, Mike Hancock From owner-freebsd-smp Sun Dec 15 19:05:35 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id TAA03023 for smp-outgoing; Sun, 15 Dec 1996 19:05:35 -0800 (PST) Received: from po2.glue.umd.edu (root@po2.glue.umd.edu [129.2.128.45]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id TAA03016 for ; Sun, 15 Dec 1996 19:05:28 -0800 (PST) Received: from gilligan.eng.umd.edu (gilligan.eng.umd.edu [129.2.103.21]) by po2.glue.umd.edu (8.8.3/8.7.3) with ESMTP id WAA07605; Sun, 15 Dec 1996 22:05:25 -0500 (EST) Received: from localhost (chuckr@localhost) by gilligan.eng.umd.edu (8.8.3/8.7.3) with SMTP id WAA13567; Sun, 15 Dec 1996 22:05:24 -0500 (EST) X-Authentication-Warning: gilligan.eng.umd.edu: chuckr owned process doing -bs Date: Sun, 15 Dec 1996 22:05:24 -0500 (EST) From: Chuck Robey X-Sender: chuckr@gilligan.eng.umd.edu To: Michael Hancock cc: smp@freebsd.org Subject: Re: General SMP Design In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Mon, 16 Dec 1996, Michael Hancock wrote: > This paper was brought up on hackers and I think it has some good > guidelines for SMP design. I am not advocating Non-blocking > Synchronization yet, because I still don't know enough about it, but much > of what is written is good for SMP in general. > > http://www-dsg.stanford.edu/papers/non-blocking-osdi/index.html > > We have many examples of type-stable memory management in our code. i.e. > a vnode is always an instance of a vnode, it can be a on a free list. It > doesn't change it's type. The zone allocator is a good thing. I read that, but I don't clearly see why TSM was required. Not talking about the benefits, but why it's required by NBS. Are you clearer, and could you explain it without too much verbage (I don't want to make everyone read a thesis here 8-> ). > > The other interesting things aside from NBS included CMDS, contention > minimizing data structures. > > They advocate per-processor pools which many agree is good for SMP > performance. > > I've noticed a lot of headers regarding TLB shootdowns recently. The text > "Unix Internals: The New Frontiers" by Uresh Vahalia has some algorithms. > > Regards, > > > Mike Hancock > > ----------------------------+----------------------------------------------- Chuck Robey | Interests include any kind of voice or data chuckr@eng.umd.edu | communications topic, C programming, and Unix. 9120 Edmonston Ct #302 | Greenbelt, MD 20770 | I run Journey2 and picnic, both FreeBSD (301) 220-2114 | version 3.0 current -- and great FUN! ----------------------------+----------------------------------------------- From owner-freebsd-smp Sun Dec 15 19:28:36 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id TAA03689 for smp-outgoing; Sun, 15 Dec 1996 19:28:36 -0800 (PST) Received: from parkplace.cet.co.jp (parkplace.cet.co.jp [202.32.64.1]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id TAA03684 for ; Sun, 15 Dec 1996 19:28:33 -0800 (PST) Received: from localhost (michaelh@localhost) by parkplace.cet.co.jp (8.8.3/CET-v2.1) with SMTP id DAA24963; Mon, 16 Dec 1996 03:28:26 GMT Date: Mon, 16 Dec 1996 12:28:26 +0900 (JST) From: Michael Hancock Reply-To: Michael Hancock To: Chuck Robey cc: smp@freebsd.org Subject: Re: General SMP Design In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Sun, 15 Dec 1996, Chuck Robey wrote: > On Mon, 16 Dec 1996, Michael Hancock wrote: > > > http://www-dsg.stanford.edu/papers/non-blocking-osdi/index.html > > > > We have many examples of type-stable memory management in our code. i.e. > > a vnode is always an instance of a vnode, it can be a on a free list. It > > doesn't change it's type. The zone allocator is a good thing. > > I read that, but I don't clearly see why TSM was required. Not talking > about the benefits, but why it's required by NBS. Are you clearer, and > could you explain it without too much verbage (I don't want to make > everyone read a thesis here 8-> ). I'm still reading it myself, but I think it is because you don't want a data structure to change "underneath" you while other higher priority processes take over your lock and you retry later on. Type-stable memory simplifies implementation. SMP is complex, which why I think people should read good background material. Good infrastructure design can reduce complexity and improve performance at the same time. "UNIX Systems for modern architectures" by Schimmel and "UNIX Internals: the new frontiers" by Vahalia have the traditional approaches. When reading that stuff, you can't help wondering if there's a better way to do solve priority inversion, deadlocks, and all those other SMP problems. I don't know if NBS is a real silver bullet, but it's cool that find someone out there thinking about alternatives to what we know today. Regards, Mike Hancock From owner-freebsd-smp Sun Dec 15 19:44:16 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id TAA04648 for smp-outgoing; Sun, 15 Dec 1996 19:44:16 -0800 (PST) Received: from po1.glue.umd.edu (root@po1.glue.umd.edu [129.2.128.44]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id TAA04643 for ; Sun, 15 Dec 1996 19:44:13 -0800 (PST) Received: from gilligan.eng.umd.edu (gilligan.eng.umd.edu [129.2.103.21]) by po1.glue.umd.edu (8.8.3/8.7.3) with ESMTP id WAA10725; Sun, 15 Dec 1996 22:44:10 -0500 (EST) Received: from localhost (chuckr@localhost) by gilligan.eng.umd.edu (8.8.3/8.7.3) with SMTP id WAA13626; Sun, 15 Dec 1996 22:44:09 -0500 (EST) X-Authentication-Warning: gilligan.eng.umd.edu: chuckr owned process doing -bs Date: Sun, 15 Dec 1996 22:44:09 -0500 (EST) From: Chuck Robey X-Sender: chuckr@gilligan.eng.umd.edu To: Michael Hancock cc: smp@freebsd.org Subject: Re: General SMP Design In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Mon, 16 Dec 1996, Michael Hancock wrote: > On Sun, 15 Dec 1996, Chuck Robey wrote: > > > On Mon, 16 Dec 1996, Michael Hancock wrote: > > > > > http://www-dsg.stanford.edu/papers/non-blocking-osdi/index.html > > > > > > We have many examples of type-stable memory management in our code. i.e. > > > a vnode is always an instance of a vnode, it can be a on a free list. It > > > doesn't change it's type. The zone allocator is a good thing. > > > > I read that, but I don't clearly see why TSM was required. Not talking > > about the benefits, but why it's required by NBS. Are you clearer, and > > could you explain it without too much verbage (I don't want to make > > everyone read a thesis here 8-> ). > > I'm still reading it myself, but I think it is because you don't want a > data structure to change "underneath" you while other higher priority > processes take over your lock and you retry later on. Type-stable memory > simplifies implementation. > > SMP is complex, which why I think people should read good background > material. Good infrastructure design can reduce complexity and improve > performance at the same time. > > "UNIX Systems for modern architectures" by Schimmel and "UNIX Internals: > the new frontiers" by Vahalia have the traditional approaches. When > reading that stuff, you can't help wondering if there's a better way to do > solve priority inversion, deadlocks, and all those other SMP problems. > > I don't know if NBS is a real silver bullet, but it's cool that find > someone out there thinking about alternatives to what we know today. I thought the URL you posted above was the thesis (which I'd already downloaded and read) but what you gave is an expanded form of it, so I have to study it and see if maybe the TSM concept is explained more fully. Thanks. > > Regards, > > > Mike Hancock > > ----------------------------+----------------------------------------------- Chuck Robey | Interests include any kind of voice or data chuckr@eng.umd.edu | communications topic, C programming, and Unix. 9120 Edmonston Ct #302 | Greenbelt, MD 20770 | I run Journey2 and picnic, both FreeBSD (301) 220-2114 | version 3.0 current -- and great FUN! ----------------------------+----------------------------------------------- From owner-freebsd-smp Sun Dec 15 21:06:23 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id VAA06891 for smp-outgoing; Sun, 15 Dec 1996 21:06:23 -0800 (PST) Received: from cs.utah.edu (cs.utah.edu [128.110.4.21]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id VAA06886 for ; Sun, 15 Dec 1996 21:06:19 -0800 (PST) Received: from fast.cs.utah.edu by cs.utah.edu (8.8.4/utah-2.21-cs) id WAA09618; Sun, 15 Dec 1996 22:06:13 -0700 (MST) Received: by fast.cs.utah.edu (8.6.10/utah-2.15-leaf) id WAA18310; Sun, 15 Dec 1996 22:06:07 -0700 Date: Sun, 15 Dec 1996 22:06:07 -0700 From: vanmaren@fast.cs.utah.edu (Kevin Van Maren) Message-Id: <199612160506.WAA18310@fast.cs.utah.edu> To: chuckr@Glue.umd.edu, michaelh@cet.co.jp Subject: Re: General SMP Design Cc: smp@freebsd.org Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Without Type Stable Memory, you can't just traverse a data structure that is getting modified under you. If someone deletes an element in a linked list while you are "looking at it", and re-mallocs it as something else ... you can go off into la-la land. With type- stable memory, it will always be an element in the list; if it gets modified you fail when you try to modify the structure and you start over again. From owner-freebsd-smp Mon Dec 16 02:32:28 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id CAA17111 for smp-outgoing; Mon, 16 Dec 1996 02:32:28 -0800 (PST) Received: from rodan.UU.NET (84@rodan.UU.NET [153.39.130.10]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id CAA17090 for ; Mon, 16 Dec 1996 02:32:24 -0800 (PST) Received: by rodan.UU.NET id QQbuic14582; Mon, 16 Dec 1996 05:32:23 -0500 (EST) Date: Mon, 16 Dec 1996 05:32:23 -0500 (EST) From: mo@UU.NET (Mike O'Dell) Message-Id: To: smp@freebsd.org Subject: smp technology..... Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk just to toss in my 2 cents.... i assume folks have read the smp paper that appeared in Computing Systems several years ago by the folks at Amdahl?? they did an smp implementation where most of the work was done by transforming the SPLn() calls (essentially with a SED script) and then getting the interrupt stuff right. they compared their experimental implementation with a fine-grain-locking implementation and discovered that the experimental system had better large-scale multiprogramming throughput because it did did significantly less convoying. (convoying is a phenomenon where processes line up and progress through a series of locks single-file like a convoy. i always thought a conga-line was a better visual image, but "convoying" is the term in the literature.) probably worth looking up the Computing Systems paper if you haven't seen it. cheers, -mo From owner-freebsd-smp Mon Dec 16 03:49:07 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id DAA20811 for smp-outgoing; Mon, 16 Dec 1996 03:49:07 -0800 (PST) Received: from hda.hda.com (ip53-max1-fitch.ziplink.net [199.232.245.53]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id DAA20806 for ; Mon, 16 Dec 1996 03:49:02 -0800 (PST) Received: (from dufault@localhost) by hda.hda.com (8.6.12/8.6.12) id GAA19402; Mon, 16 Dec 1996 06:46:20 -0500 From: Peter Dufault Message-Id: <199612161146.GAA19402@hda.hda.com> Subject: Re: General SMP Design In-Reply-To: from Chuck Robey at "Dec 15, 96 10:44:09 pm" To: chuckr@Glue.umd.edu (Chuck Robey) Date: Mon, 16 Dec 1996 06:46:20 -0500 (EST) Cc: michaelh@cet.co.jp, smp@freebsd.org X-Mailer: ELM [version 2.4ME+ PL25 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > I thought the URL you posted above was the thesis (which I'd already > downloaded and read) but what you gave is an expanded form of it, so I > have to study it and see if maybe the TSM concept is explained more fully. > Thanks. A question for those who know the Intel SMP system: I was looking at the work at UNC that Bakul pointed out. Some of the work assumes that a lower priority process will never complete before a higher priority process leading to uniprocessor restrictions. Is there an efficient way to have a priority on the processor pool to permit only a single process at priority N (and as many that want to at a higher priority) to run? This would not be going on in general, but only during the DWCAS/READ operations. Peter -- Peter Dufault (dufault@hda.com) Real-Time Machine Control and Simulation HD Associates, Inc. Voice: 508 433 6936 From owner-freebsd-smp Mon Dec 16 04:43:03 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id EAA24808 for smp-outgoing; Mon, 16 Dec 1996 04:43:03 -0800 (PST) Received: from whale.gu.kiev.ua (whale.gu.net [194.93.190.4]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id EAA24795 for ; Mon, 16 Dec 1996 04:42:52 -0800 (PST) Received: from creator.gu.kiev.ua (stesin@creator.gu.kiev.ua [194.93.190.3]) by whale.gu.kiev.ua (8.7.5/8.7.3) with SMTP id OAA64308 for ; Mon, 16 Dec 1996 14:42:42 +0200 Date: Mon, 16 Dec 1996 14:42:41 +0200 (EET) From: Andrew Stesin X-Sender: stesin@creator.gu.kiev.ua To: smp@freebsd.org Subject: SOYO 5TH2 motherboard Message-ID: X-NCC-RegID: ua.gu MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hello people, I'm going to deal with one of this 5TH2 beasts tomorrow; it isn't listed directly in the "supported" list, but one of the links from SMP page has a pointer to it. It will be running in 2 x boxed-P100 CPUs (boxed == brand new) configuration, with 60ns EDO RAM. And either NCR SCSI ;) or even IDE :( Can anyone provide some opinions on it before I'll jump into it tomorrow? (I'll do my best to run a FreeBSD and a MPP test program on it, but who knows if I'll be able). -- Best, Andrew Stesin nic-hdl: ST73-RIPE From owner-freebsd-smp Mon Dec 16 10:15:01 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id KAA22164 for smp-outgoing; Mon, 16 Dec 1996 10:15:01 -0800 (PST) Received: from charon.finall.com (charon.finall.com [206.246.160.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id KAA22151 for ; Mon, 16 Dec 1996 10:14:57 -0800 (PST) Received: from exchange.finall.com (exchange.finall.com [206.246.160.132]) by charon.finall.com (8.6.12/8.6.12) with SMTP id NAA05341 for ; Mon, 16 Dec 1996 13:14:47 -0500 Received: by exchange.finall.com with Microsoft Exchange (IMC 4.0.837.3) id <01BBEB53.1C254C80@exchange.finall.com>; Mon, 16 Dec 1996 13:14:46 -0500 Message-ID: From: "Jung, Michael" To: "'FreeBSD-SMP Mailing List'" Subject: Bad Register Date: Mon, 16 Dec 1996 13:14:44 -0500 X-Mailer: Microsoft Exchange Server Internet Mail Connector Version 4.0.837.3 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Using 2.2961014-SNAP to build the kernel I get the following error while running make. (Note it fails likewise when compiling GENERIC) Do I need to be using a older SNAP release? --mikej Micahel Jung (pinky.thepoint.net) $ make cc -c -x assembler-with-cpp -DLOCORE -nostdinc -I- -I. -I../.. -I/usr/include -DAPM_BROKEN_STATCLOCK -DFAILSAFE -DCOMPAT_43 -DCD9660 -DMSDOSFS -DNFS -DFFS -DINET -DKERNEL ../../i386/i386/locore.s ../../i386/i386/locore.s: Assembler messages: ../../i386/i386/locore.s:730: Error: bad register name ('%cr4') ../../i386/i386/locore.s:732: Error: bad register name ('%cr4') *** Error code 1 Stop. (pinky.thepoint.net) $ From owner-freebsd-smp Mon Dec 16 10:29:34 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id KAA23052 for smp-outgoing; Mon, 16 Dec 1996 10:29:34 -0800 (PST) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id KAA23047; Mon, 16 Dec 1996 10:29:26 -0800 (PST) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id LAA01626; Mon, 16 Dec 1996 11:27:15 -0700 From: Terry Lambert Message-Id: <199612161827.LAA01626@phaeton.artisoft.com> Subject: Re: some questions concerning TLB shootdowns in FreeBSD To: toor@dyson.iquest.net (John S. Dyson) Date: Mon, 16 Dec 1996 11:27:14 -0700 (MST) Cc: terry@lambert.org, toor@dyson.iquest.net, phk@critter.tfs.com, peter@spinner.dialix.com, dyson@freebsd.org, smp@freebsd.org, haertel@ichips.intel.com In-Reply-To: <199612160058.TAA05793@dyson.iquest.net> from "John S. Dyson" at Dec 15, 96 07:58:05 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Terry, I am NOT discounting your suggestions -- but I am > bringing up challenges associated with your suggestions. OK... 8-). > > Some potential optimizations: > > > > 1) This only applys to written pages not marked copy-on-write; > > read-only pages and pages that will be copied on write (like > > those in your note about "address spaces are already shared") > > You still need to be able to globally invalidate pages that are mapped > read-only. (e.g. paging out (where things can happen anytime, or > object reclaimation -- that of course doesn't happen unless an object > is done with.)) OK, I buy this one. Not only would it have to be done with, it would have to be at the end of the LRU to get it forced out, right? In that particular case, it might be worthwhile to invalidate down to a low water mark of unreferenced pages when a reclaim is necessary. > > 2) Flushing can be "lazy" in most cases. That is, the page could > > be marked invalid for a particular CPU, and only flushed if > > that CPU needs to use it. For a generic first time implementation, > > a single unsigned long with CPU invalidity bits could be used > > (first time because it places a 32 processor limit, which I > > feel is an unacceptable limitation -- I want to run on Connection > > Machines some day) as an addition to the page attributes. For > > the most part, it is important to realize that this is a > > negative validity indicator. This dictates who has to do > > the work: the CPU that wants to access the page. The higher > > the process CPU affinity, the less this will happen. > > There is no special indication of a TLB entry being updated in a > processor from the page tables. So, once there is a page table > entry created, we have no indication when the processor grabs > it (I seem to remember that there are ways for coercing P6's > and perhaps P5's into doing it though.) The only way that I > would try to do it is to get information from Intel saying that > the method is "blessed." It could break things for other (non-Intel) > CPU's though. By default, you mark all cpu's but the one doing the initial mapping as "invalid". If the execution context needing the page stays on that CPU, you never have to update anyone. Only when you move an execution context between CPU's do you have to update, and then only the target CPU (and potentially preinvalidate those on the source CPU). Shared pages are more problematic, but at least only the CPU's with mapping bits turned off need to participate in the IPI hold. This presumes some (minor) scheduler hooks with per CPU run queues, which I admit is slightly different that what there is now. The scheduler would only choose to move a process between queues based on thread non-affinity (ie: multiple kernel threads trying for SMP scalability) or to load-balance the overall system. In both cases, it knows about the mapping on the source CPU and so can tell the target CPU about the mappings. The point is that the mappings on all CPU's need not be identical, only valid for the execution contexts in that CPU's run queue. You consider the kernel itself as a single shared execution context, but the kernel mapping only has to be done once, up front, at load time, or again at module load time. As CPU's destroy page references, they must hold a mutex (or the reference counts, which are only examined on create/destroy/change_CPU may be placed in non-cacheable memory). If the bitmap goes to zero references, the CPU doing the decrement returns it to the global pool. Is there a reason that the CPU destroying it's last reference would not be able to invalidate for itself at that time? Are you concerned that what is needed is "non-L2 cacheable"? If so, I don't think it's necessary. If it is necessary, then you're right, it would cause problems on other CPUs... the BeBox, a dual PPC603 machine, adds the second CPU in place of an L2 cache. As a result, it only supports MEI coherency protocol, *not* MESI. This would be a problem. On the other hand, it's like soft updates: you implement the calls at the synchronization nodes in the event graph, and for a trivial implementation, you synchronously complete the call at the time the call is made instead of when the first cycle would occur in the graph element list. For processors that support TLB update, you encapsulate the "blessed" code. For those that don't, you do the update immediately. The code making the call doesn't know what the macro will make it do on a given processor; the macro is only a synchronization point, and the act of resolving the synchronization itself is CPU specific (and therefore opaque). Personally, I don't think TLB update is an issue for multiple CPU's as long as there are per CPU reference counts and a global bitmap that indicates that a given CPU has no reference counts on it (the point being to avoid locking and to hide the non-cacheable references in the expensive operations you already have to do -- even the per CPU count can be cacheable, since it's private to that CPU). Regards, Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Mon Dec 16 10:32:33 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id KAA23168 for smp-outgoing; Mon, 16 Dec 1996 10:32:33 -0800 (PST) Received: from who.cdrom.com (who.cdrom.com [204.216.27.3]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id KAA23163 for ; Mon, 16 Dec 1996 10:32:30 -0800 (PST) Received: from cs.utah.edu (cs.utah.edu [128.110.4.21]) by who.cdrom.com (8.7.5/8.6.11) with ESMTP id KAA26262 for ; Mon, 16 Dec 1996 10:32:27 -0800 (PST) Received: from fast.cs.utah.edu by cs.utah.edu (8.8.4/utah-2.21-cs) id LAA20440; Mon, 16 Dec 1996 11:31:05 -0700 (MST) Received: by fast.cs.utah.edu (8.6.10/utah-2.15-leaf) id LAA06197; Mon, 16 Dec 1996 11:30:54 -0700 Date: Mon, 16 Dec 1996 11:30:54 -0700 From: vanmaren@fast.cs.utah.edu (Kevin Van Maren) Message-Id: <199612161830.LAA06197@fast.cs.utah.edu> To: mikej@finall.com, smp@freebsd.org Subject: Re: Bad Register Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk You need a newer version of the binutils (ala gas). From owner-freebsd-smp Mon Dec 16 10:33:25 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id KAA23216 for smp-outgoing; Mon, 16 Dec 1996 10:33:25 -0800 (PST) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id KAA23210 for ; Mon, 16 Dec 1996 10:33:22 -0800 (PST) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id LAA01660; Mon, 16 Dec 1996 11:32:22 -0700 From: Terry Lambert Message-Id: <199612161832.LAA01660@phaeton.artisoft.com> Subject: Re: General SMP Design To: michaelh@cet.co.jp (Michael Hancock) Date: Mon, 16 Dec 1996 11:32:22 -0700 (MST) Cc: smp@freebsd.org In-Reply-To: from "Michael Hancock" at Dec 16, 96 11:47:20 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > The other interesting things aside from NBS included CMDS, contention > minimizing data structures. > > They advocate per-processor pools which many agree is good for SMP > performance. It's more an issue of context migration and synchronization, and the impact on scalability. Many people who do not consider scalability (ie: a limit of 4 processors is acceptable to them) don't understand that the limit is imposed by synchronization arbitration. Per processor pools increase the number of processors you can do without hitting this limitation. For 2-4 processors, this limitation really does not impact performance enough to notice. After 4 processors, however, it becomes a real issue. So it's a post-scaling performance issue, or a scalability issue more than a pure performance issue. Regards, Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Mon Dec 16 10:43:27 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id KAA23722 for smp-outgoing; Mon, 16 Dec 1996 10:43:27 -0800 (PST) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id KAA23717 for ; Mon, 16 Dec 1996 10:43:24 -0800 (PST) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id LAA01687; Mon, 16 Dec 1996 11:41:52 -0700 From: Terry Lambert Message-Id: <199612161841.LAA01687@phaeton.artisoft.com> Subject: Re: General SMP Design To: michaelh@cet.co.jp Date: Mon, 16 Dec 1996 11:41:52 -0700 (MST) Cc: chuckr@Glue.umd.edu, smp@freebsd.org In-Reply-To: from "Michael Hancock" at Dec 16, 96 12:28:26 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > > I read that, but I don't clearly see why TSM was required. Not talking > > about the benefits, but why it's required by NBS. Are you clearer, and > > could you explain it without too much verbage (I don't want to make > > everyone read a thesis here 8-> ). > > I'm still reading it myself, but I think it is because you don't want a > data structure to change "underneath" you while other higher priority > processes take over your lock and you retry later on. Type-stable memory > simplifies implementation. But it's not a magic bullet, itself. I like the "delay work hoping it will go away" approach to NBS... I've advocated locking most resources in CPU specific pools, and using the migration of resource between processors, and high and low water marks on the pools to allow them to refill from, or drain to, global pools which are only used for resource balancing. NBS is an alternate approach. I kind of dislike resource preemption as a means of solving priority inversion. I may have meta-state which I need to hold. I suppose if the preemption were read-only and TSM'ed at the meta-state level, I could live with it. Otherwise, I prefer priority lending, even though there are some obvious loopholes in fairness that it can open up. Bottom line is that lending only works well if the resource is only held at most for the duration of the kernel entrancy. If this is obeyed, and kernel preemption exists (so that when the resource is released, the priority is immediately unlent and the process lent is descheduled), then there is really no advantage to NBS over a CPU centric hierarchical locking method and global pooling. Regards, Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Mon Dec 16 11:12:35 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id LAA25458 for smp-outgoing; Mon, 16 Dec 1996 11:12:35 -0800 (PST) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id LAA25453 for ; Mon, 16 Dec 1996 11:12:33 -0800 (PST) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id MAA01775; Mon, 16 Dec 1996 12:11:40 -0700 From: Terry Lambert Message-Id: <199612161911.MAA01775@phaeton.artisoft.com> Subject: Re: smp technology..... To: mo@UU.NET (Mike O'Dell) Date: Mon, 16 Dec 1996 12:11:40 -0700 (MST) Cc: smp@freebsd.org In-Reply-To: from "Mike O'Dell" at Dec 16, 96 05:32:23 am X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > i assume folks have read the smp paper that appeared in Computing Systems > several years ago by the folks at Amdahl?? they did an smp implementation > where most of the work was done by transforming the SPLn() calls > (essentially with a SED script) and then getting the interrupt stuff > right. > > they compared their experimental implementation with a fine-grain-locking > implementation and discovered that the experimental system had better > large-scale multiprogramming throughput because it did did significantly > less convoying. (convoying is a phenomenon where processes line up > and progress through a series of locks single-file like a convoy. > i always thought a conga-line was a better visual image, but > "convoying" is the term in the literature.) "Several years ago" says "SVR4.0.2 ES/MP" or "Sequent" to me... both of these have acknowledged concurrency problems in their device access, and even in their FS code itself. If this was the comparison, I'd like to see another one against UnixWare 2.x (forget Solaris; Sun did their FS synchronization at the wrong level). I suspect it would not do nearly so well (unless they leap-frogged UnixWare's UFS's DOW with soft updates or something). Regards, Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. From owner-freebsd-smp Mon Dec 16 11:42:59 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id LAA27591 for smp-outgoing; Mon, 16 Dec 1996 11:42:59 -0800 (PST) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id LAA27580 for ; Mon, 16 Dec 1996 11:42:53 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id MAA29994; Mon, 16 Dec 1996 12:42:40 -0700 Message-Id: <199612161942.MAA29994@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: "Jung, Michael" cc: "'FreeBSD-SMP Mailing List'" Subject: Re: Bad Register In-reply-to: Your message of "Mon, 16 Dec 1996 13:14:44 EST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 16 Dec 1996 12:42:40 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, > Using 2.2961014-SNAP to build the kernel I get the following error while > running make. (Note it fails likewise when compiling GENERIC) > > Do I need to be using a older SNAP release? you probably should cvsup -current, then make world, then attempt to build the SMP specific sys tree. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Mon Dec 16 18:19:53 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id SAA25700 for smp-outgoing; Mon, 16 Dec 1996 18:19:53 -0800 (PST) Received: from charon.finall.com (charon.finall.com [206.246.160.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id SAA25652 for ; Mon, 16 Dec 1996 18:19:48 -0800 (PST) Received: from exchange.finall.com (exchange.finall.com [206.246.160.132]) by charon.finall.com (8.6.12/8.6.12) with SMTP id VAA07909 for ; Mon, 16 Dec 1996 21:19:41 -0500 Received: by exchange.finall.com with Microsoft Exchange (IMC 4.0.837.3) id <01BBEB96.DD073740@exchange.finall.com>; Mon, 16 Dec 1996 21:19:46 -0500 Message-ID: From: "Jung, Michael" To: "'FreeBSD SMP Mailing List'" Subject: Undefined Symbol Date: Mon, 16 Dec 1996 21:19:44 -0500 X-Mailer: Microsoft Exchange Server Internet Mail Connector Version 4.0.837.3 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Got the new assembler compiled (fixed unknown register - tnx).. During make the this error occurs..... loading kernel support.o: Undefined symbol `_curpcb' referenced from text segment support.o: Undefined symbol `_curpcb' referenced from text segment support.o: Undefined symbol `_curpcb' referenced from text segment support.o: Undefined symbol `_curpcb' referenced from text segment support.o: Undefined symbol `_curpcb' referenced from text segment support.o: Undefined symbol `_curpcb' referenced from text segment support.o: Undefined symbol `_curpcb' referenced from text segment support.o: Undefined symbol `_curpcb' referenced from text segment support.o: Undefined symbol `_curpcb' referenced from text segment support.o: More undefined symbol _curpcb refs follow *** Error code 1 Stop. (pinky) /usr/src/sys/compile/SMP # Are their rev levels that can be determined from CVS when reporting problems like this (if that is a issue)? cvsup current as of ~21:20 Dec 16 Thanks again --mikej Michael Jung Sr. Systems/Network Administrator Financial Alliance Processing Services, Inc. mikej@finall.com From owner-freebsd-smp Mon Dec 16 18:22:04 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id SAA25988 for smp-outgoing; Mon, 16 Dec 1996 18:22:04 -0800 (PST) Received: from parkplace.cet.co.jp (parkplace.cet.co.jp [202.32.64.1]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id SAA25983 for ; Mon, 16 Dec 1996 18:22:02 -0800 (PST) Received: from localhost (michaelh@localhost) by parkplace.cet.co.jp (8.8.3/CET-v2.1) with SMTP id CAA01102; Tue, 17 Dec 1996 02:20:40 GMT Date: Tue, 17 Dec 1996 11:20:40 +0900 (JST) From: Michael Hancock To: haertel@ichips.intel.com cc: smp@freebsd.org Subject: Re: some questions concerning TLB shootdowns in FreeBSD In-Reply-To: <9612141725.AA57406@pdxcs078.intel.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Sat, 14 Dec 1996 haertel@ichips.intel.com wrote: > 1. IPI to everyone sharing these specific PTE's > 2. wait at barrier until everyone arrives > 3. manipulate PTE > 4. release barrier > 5. everyone (including us) flushes TLB's Here's SVR4.2/MP's algorithm for x86 from Unix Internals with terminology translated to what we use: Initiator (Shooter) ------------------- 1. lock pmap acquire global spinlock to prevent other shootdowns 2. send IPI and wait for ack 3. modify ptes 4. increment counter 5. flush own tlb unlock global spinlock unlock pmap Responder --------- 1. receive IPI 2. wait for counter to be incremented 3. flush own tlb In SVR4.2/MP the hat (pmap) has a list of processors that the process could be active on. I don't know if we can make some of the same assumptions for optimizations in FreeBSD, but we can probably find some later. Places to look... 1) Look at all the FreeBSD events that cause tlbs and look for inherent synchronization in the sequence. 2) Look at kernel vs. user tlb invalidations. SVR4.2/MP can guarantee that the kernel won't access an invalid tlb so can it can delay shootdowns and hope that some other event such as a context switch will flush it anyway. 3) Intel specific optimizations. For example it has a level-1 (pointed to by CR3/PDBR) page table containing ptes for level-2 tables. In the multiple pte case it can just invalidate appropriate entries in the level-1 table. The messages here have been leaving me in the dust, I'm sorry if I've said something that's already been discussed. Regards, Mike From owner-freebsd-smp Mon Dec 16 19:09:51 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id TAA21633 for smp-outgoing; Mon, 16 Dec 1996 19:09:51 -0800 (PST) Received: from dyson.iquest.net (dyson.iquest.net [198.70.144.127]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id TAA21625 for ; Mon, 16 Dec 1996 19:09:48 -0800 (PST) Received: (from root@localhost) by dyson.iquest.net (8.8.2/8.6.9) id WAA01242; Mon, 16 Dec 1996 22:09:33 -0500 (EST) From: "John S. Dyson" Message-Id: <199612170309.WAA01242@dyson.iquest.net> Subject: Re: Bad Register To: mikej@finall.com (Jung, Michael) Date: Mon, 16 Dec 1996 22:09:33 -0500 (EST) Cc: smp@freebsd.org In-Reply-To: from "Jung, Michael" at Dec 16, 96 01:14:44 pm Reply-To: dyson@freebsd.org X-Mailer: ELM [version 2.4 PL24 ME8] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Using 2.2961014-SNAP to build the kernel I get the following error while > running make. (Note it fails likewise when compiling GENERIC) > > Do I need to be using a older SNAP release? > Nope, but you need to update your assembler to something newer. John From owner-freebsd-smp Mon Dec 16 19:11:35 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id TAA21729 for smp-outgoing; Mon, 16 Dec 1996 19:11:35 -0800 (PST) Received: from who.cdrom.com (who.cdrom.com [204.216.27.3]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id TAA21705 for ; Mon, 16 Dec 1996 19:11:31 -0800 (PST) Received: from avatar.avatar.com (avatar.avatar.com [199.33.206.17]) by who.cdrom.com (8.7.5/8.6.11) with ESMTP id SAA26893 for ; Mon, 16 Dec 1996 18:45:56 -0800 (PST) Received: from avatar.avatar.com (kory@avatar.avatar.com [199.33.206.17]) by avatar.avatar.com (8.7.4/8.6.9) with SMTP id SAA26919 for ; Mon, 16 Dec 1996 18:45:23 -0800 (PST) Date: Mon, 16 Dec 1996 18:45:22 -0800 (PST) From: Kory Hamzeh To: freebsd-smp@FreeBSD.ORG Subject: AMI Merlin DP Motherboard Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk I'm about to take a plung and buy the AMI Merlin DP Dual Pentium Pro motherboard (my second choice is the SuperMicro P6DOF). Has anyone tired running the SMP kernel on this puppy yet? Is there any reason I should not buy this MB? I know the SMP kernel is not very stable on P6 yet, and I know it will take several more month before it happens. Thanks, Kory From owner-freebsd-smp Mon Dec 16 19:30:44 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id TAA22629 for smp-outgoing; Mon, 16 Dec 1996 19:30:44 -0800 (PST) Received: from bluenose.na.tuns.ca (bluenose.na.tuns.ca [134.190.50.156]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id TAA22623 for ; Mon, 16 Dec 1996 19:30:34 -0800 (PST) Received: (from smp@localhost) by bluenose.na.tuns.ca (8.7.6/8.7.3) id XAA00488; Mon, 16 Dec 1996 23:34:40 -0400 (AST) From: "J.M. Chuang" Message-Id: <199612170334.XAA00488@bluenose.na.tuns.ca> Subject: Re: Undefined Symbol To: mikej@finall.com (Jung, Michael) Date: Mon, 16 Dec 1996 23:34:40 -0400 (AST) Cc: smp@freebsd.org In-Reply-To: from "Jung, Michael" at "Dec 16, 96 09:19:44 pm" X-Mailer: ELM [version 2.4ME+ PL13 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk > Got the new assembler compiled (fixed unknown register - tnx).. > During make the this error occurs..... > > loading kernel > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: Undefined symbol `_curpcb' referenced from text segment > support.o: More undefined symbol _curpcb refs follow > *** Error code 1 > > Stop. > (pinky) /usr/src/sys/compile/SMP # > > Are their rev levels that can be determined from CVS when reporting > problems like this (if that is a issue)? cvsup current as of ~21:20 Dec > 16 > You have to add two extra lines in your config file (SMP): options SMP options NCPU=2 It seems that smp-kernel source does not like the config file without these two lines. Jim From owner-freebsd-smp Mon Dec 16 20:16:34 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id UAA24943 for smp-outgoing; Mon, 16 Dec 1996 20:16:34 -0800 (PST) Received: from rodan.UU.NET (84@rodan.UU.NET [153.39.130.10]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id UAA24938 for ; Mon, 16 Dec 1996 20:16:30 -0800 (PST) Received: from localhost by rodan.UU.NET with SMTP (peer crosschecked as: mo@localhost) id QQbukv05044; Mon, 16 Dec 1996 23:15:00 -0500 (EST) Message-Id: To: Terry Lambert cc: mo@UU.NET (Mike O'Dell), smp@freebsd.org Subject: Re: smp technology..... In-reply-to: Your message of "Mon, 16 Dec 1996 12:11:40 MST." <199612161911.MAA01775@phaeton.artisoft.com> Date: Mon, 16 Dec 1996 23:14:59 -0500 From: "Mike O'Dell" Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk no, it is not SVR4xxx it was for a berkeley kernel. and it was amdahl. consult the usenix online index -mo From owner-freebsd-smp Mon Dec 16 21:43:58 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id VAA00599 for smp-outgoing; Mon, 16 Dec 1996 21:43:58 -0800 (PST) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id VAA00587 for ; Mon, 16 Dec 1996 21:43:55 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id WAA02920; Mon, 16 Dec 1996 22:43:38 -0700 Message-Id: <199612170543.WAA02920@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: "J.M. Chuang" cc: mikej@finall.com (Jung, Michael), smp@freebsd.org Subject: Re: Undefined Symbol In-reply-to: Your message of "Mon, 16 Dec 1996 23:34:40 -0400." <199612170334.XAA00488@bluenose.na.tuns.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 16 Dec 1996 22:43:38 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, >You have to add two extra lines in your config file (SMP): > >options SMP >options NCPU=2 > >It seems that smp-kernel source does not like the config file without these >two lines. this used to be true, now you only need the 1st: "options SMP". the others are all set to reasonable defaults if not explictetly set. mptable will give you a complete list of SMP related options, 'optimized' for your particular hardware. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Mon Dec 16 21:49:47 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id VAA00773 for smp-outgoing; Mon, 16 Dec 1996 21:49:47 -0800 (PST) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id VAA00768 for ; Mon, 16 Dec 1996 21:49:43 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id WAA02970; Mon, 16 Dec 1996 22:49:33 -0700 Message-Id: <199612170549.WAA02970@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Kory Hamzeh cc: freebsd-smp@FreeBSD.ORG Subject: Re: AMI Merlin DP Motherboard In-reply-to: Your message of "Mon, 16 Dec 1996 18:45:22 PST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 16 Dec 1996 22:49:33 -0700 Sender: owner-smp@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk Hi, > > I'm about to take a plung and buy the AMI Merlin DP Dual Pentium Pro > motherboard (my second choice is the SuperMicro P6DOF). Has anyone tired > running the SMP kernel on this puppy yet? Is there any reason I should > not buy this MB? I know the SMP kernel is not very stable on P6 yet, and > I know it will take several more month before it happens. I haven't received any reports on this board. In general AMI makes good hardware. Is it an EISA motherboard? I would caution against it if so. In general EISA boards can give grief under the SMP spec even if you don't use EISA cards. Unfortunately the spec allows some pretty braindead behaviour for EISA systems. Some are fine, it depends on the chipset and design used by the MB manufacturer. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Mon Dec 16 22:10:17 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id WAA01477 for smp-outgoing; Mon, 16 Dec 1996 22:10:17 -0800 (PST) Received: from avatar.avatar.com (avatar.avatar.com [199.33.206.17]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id WAA01472 for ; Mon, 16 Dec 1996 22:10:14 -0800 (PST) Received: from avatar.avatar.com (kory@avatar.avatar.com [199.33.206.17]) by avatar.avatar.com (8.7.4/8.6.9) with SMTP id WAA03241; Mon, 16 Dec 1996 22:09:40 -0800 (PST) Date: Mon, 16 Dec 1996 22:09:40 -0800 (PST) From: Kory Hamzeh To: Steve Passe cc: freebsd-smp@FreeBSD.ORG Subject: Re: AMI Merlin DP Motherboard In-Reply-To: <199612170549.WAA02970@clem.systemsix.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@FreeBSD.ORG X-Loop: FreeBSD.org Precedence: bulk On Mon, 16 Dec 1996, Steve Passe wrote: > Hi, > > > > > I'm about to take a plung and buy the AMI Merlin DP Dual Pentium Pro > > motherboard (my second choice is the SuperMicro P6DOF). Has anyone tired > > running the SMP kernel on this puppy yet? Is there any reason I should > > not buy this MB? I know the SMP kernel is not very stable on P6 yet, and > > I know it will take several more month before it happens. > > I haven't received any reports on this board. In general AMI makes good > hardware. Is it an EISA motherboard? I would caution against it if so. > In general EISA boards can give grief under the SMP spec even if you > don't use EISA cards. Unfortunately the spec allows some pretty braindead > behaviour for EISA systems. Some are fine, it depends on the chipset > and design used by the MB manufacturer. > Hi Steve, The AMI Merlin DP is a PCI system. It uses the Intel 440FX chipset. In the past, I've always had good luck with AMI stuff. I would prefer to use the AMI MB instead of the SuperMicro. SuperMicro customer support is truly pathetic. Thanks, Kory From owner-freebsd-smp Tue Dec 17 06:06:22 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id GAA25010 for smp-outgoing; Tue, 17 Dec 1996 06:06:22 -0800 (PST) Received: from key.sms.fi (root@key.sms.fi [194.197.125.241]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id GAA25002 for ; Tue, 17 Dec 1996 06:06:15 -0800 (PST) Received: (from count@localhost) by key.sms.fi (8.8.4/8.7.3) id QAA05468 for freebsd-smp@freebsd.org; Tue, 17 Dec 1996 16:06:12 +0200 (EET) From: "Bror 'Count' Heinola" Message-Id: <199612171406.QAA05468@key.sms.fi> Subject: Success! ...well, not really To: freebsd-smp@freebsd.org Date: Tue, 17 Dec 1996 16:06:12 +0200 (EET) X-Mailer: ELM [version 2.4ME+ PL28 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk I'm a proud owner of a Tyan Titan Pro S1668 ATX dual Ppro motherboard with two PPro 180MHz CPU's. I installed 2.2-ALPHA and then cvsup'd -current on top of that, did make world, compiled a uniprocessor kernel and SMP kernel. The machine runs just fine with both kernels when the second cpu is disabled, however, when I enable the second CPU it works, but whenever I try to "make" something, it immediately bombs with a panic like this (btw, is there a way of saving a panic output in short of writing it down by hand?) : Fatal trap 12: page fault while in kernel mode cpunumber = 0 fault virtual address = 0xfffbeff4 fault code = supervisor read, page not present instruction pointer = 0x8:0xf0176c13 stack pointer = 0x10:0xefbffe94 frame pointer = 0x10:0xefbffeb0 code segment = base 0x0, limit 0xfffff, type ox1b = dpl 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 190 (sh) interrupt mask = trap number = 12 panic (cpu#0): page fault boot() called on cpu#0 I'll include the current SMP kernel config along with latest dmesg output below. FreeBSD 3.0-SMP #0: Tue Dec 17 15:18:11 EET 1996 count@bhe0.sms.fi:/usr/src/sys-SMP/compile/BHE0SMP FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 1, version: 0x00040011 cpu1 (AP): apic id: 0, version: 0x00040011 io0 (APIC): apic id: 2, version: 0x00170011 Calibrating clock(s) relative to mc146818A clock ... i8254 clock: 1193059 Hz CPU: Pentium Pro (686-class CPU) Origin = "GenuineIntel" Id = 0x617 Stepping=7 Features=0xfbff,MTRR,PGE,MCA,CMOV> real memory = 33554432 (32768K bytes) Physical memory hole(s): avail memory = 31240192 (30508K bytes) Probing for devices on PCI bus 0: chip0 rev 2 on pci0:0 chip1 rev 1 on pci0:7:0 chip2 rev 0 on pci0:7:1 vga0 rev 48 int a irq 17 on pci0:13 Freeing (NOT implimented) irq 9 for ISA cards. ahc0 rev 0 int a irq 16 on pci0:14 Freeing (NOT implimented) irq 11 for ISA cards. ahc0: aic7880 Single Channel, SCSI Id=7, 16 SCBs ahc0: target 0 Tagged Queuing Device (ahc0:0:0): "IBM DORS-32160 !# WA3E" type 0 fixed SCSI 2 sd0(ahc0:0:0): Direct-Access 2063MB (4226725 512 byte sectors) (ahc0:6:0): "TOSHIBA CD-ROM XM-5401TA 3115" type 5 removable SCSI 2 cd0(ahc0:6:0): CD-ROM cd0(ahc0:6:0): NOT READY asc:4,1 cd0(ahc0:6:0): Logical unit is in process of becoming ready can't get the size Probing for devices on the ISA bus: sc0 at 0x60-0x6f irq 1 on motherboard sc0: VGA color <16 virtual consoles, flags=0x0> sio0 at 0x3f8-0x3ff irq 4 on isa sio0: type 16550A sio1 at 0x2f8-0x2ff irq 3 on isa sio1: type 16550A lpt0 at 0x378-0x37f irq 7 on isa lpt0: Interrupt-driven port lp0: TCP/IP capable interface psm0 at 0x60-0x64 irq 12 on motherboard psm0: device ID 0, 3 buttons? fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa fdc0: NEC 72065B fd0: 1.44MB 3.5in 1 3C5x9 board(s) on ISA found at 0x300 ep0 at 0x300-0x30f irq 10 on isa ep0: aui/utp/bnc[*UTP*] address 00:20:af:43:1f:b0 npx0 on motherboard npx0: INT 16 interface Enabled INTs: 1, 2, 3, 4, 6, 7, 8, 10, 12, 16, imen: 0x00feea21 WARNING: / was not properly dismounted. SMP: All idle procs online. SMP: Starting 1st AP! SMP: AP CPU #1 LAUNCHED!! Starting Scheduling... SMP: TADA! CPU #1 made it into the scheduler!. SMP: All 2 CPU's are online! machine "i386" cpu "I686_CPU" ident BHESMP maxusers 10 options INET #InterNETworking options FFS #Berkeley Fast Filesystem options "CD9660" #ISO 9660 Filesystem options PROCFS #Process filesystem options "COMPAT_43" #Compatible with BSD 4.3 [KEEP THIS!] options UCONSOLE #Allow users to grab the console options USERCONFIG #boot -c editor options VISUAL_USERCONFIG #visual boot -c editor options HARDFONTS options XSERVER options "AUTO_EOI1" options "AUTO_EOI2" options DUMMY_NOPS options "MAXMEM=131072" options AHC_TAGENABLE options SMP # Symmetric MultiProcessor Kernel options APIC_IO # Symmetric (APIC) I/O options NCPU=2 # number of CPUs options NBUS=2 # number of busses options NAPIC=1 # number of IO APICs options NINTR=24 # number of INTs config kernel root on sd0 controller isa0 controller pci0 controller fdc0 at isa? port "IO_FD1" bio irq 6 drq 2 vector fdintr disk fd0 at fdc0 drive 0 controller ahc0 controller scbus0 device sd0 device cd0 #Only need one of these, the code dynamically grows device sc0 at isa? port "IO_KBD" tty irq 1 vector scintr device npx0 at isa? port "IO_NPX" irq 13 vector npxintr device sio0 at isa? port "IO_COM1" tty irq 4 vector siointr device sio1 at isa? port "IO_COM2" tty irq 3 vector siointr device lpt0 at isa? port? tty irq 7 vector lptintr device psm0 at isa? port "IO_KBD" conflicts tty irq 12 vector psmintr device ep0 at isa? port 0x300 net irq 10 vector epintr pseudo-device loop pseudo-device ether pseudo-device log pseudo-device pty 64 pseudo-device gzip # Exec gzipped a.out's pseudo-device bpfilter 4 -- Bror 'Count' Heinola # Santa Monica Software # Email: bror@sms.fi Isokaari 27 A2 # Technical Support # IRC: Count FI-00200 HELSINKI # Cisco / Ascend # NIC: BH271 +358-40-523-2171 # +358-9-613-8911 # Are you defective? From owner-freebsd-smp Tue Dec 17 07:50:32 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id HAA00623 for smp-outgoing; Tue, 17 Dec 1996 07:50:32 -0800 (PST) Received: from tis-mail.thepoint.net (tis-backup.thepoint.net [198.6.9.55]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id HAA00618 for ; Tue, 17 Dec 1996 07:50:30 -0800 (PST) Received: by tis-mail.thepoint.net with SMTP (Microsoft Exchange Server Internet Mail Connector Version 4.0.993.5) id <01BBEC08.1916D1C0@tis-mail.thepoint.net>; Tue, 17 Dec 1996 10:50:20 -0500 Message-ID: From: Arlie Davis To: "FreeBSD SMP List (E-mail)" Cc: Michael Jung Subject: As of 10AM, Dec 17, SMP is pretty solid. Date: Tue, 17 Dec 1996 10:50:18 -0500 X-Mailer: Microsoft Exchange Server Internet Mail Connector Version 4.0.993.5 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk As of 10AM, Dec 17, the current SMP kernel is pretty solid. We're running it on a relatively early SMP board, an early revision of the ASUS P54NP4, I think. Dual P90 -- nothing to write home about these days. Just wanted to say THANKS for the good work! The kernel is solid; we've been beating it to death since we turned on the second processor, and it hasn't flinched. We're running without the APIC_IO option, which I understand is more advanced but currently more risky, so this is hardly new news. We are building a kernel with APIC_IO, and will report on that, later. -- arlie From owner-freebsd-smp Tue Dec 17 09:17:58 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id JAA07413 for smp-outgoing; Tue, 17 Dec 1996 09:17:58 -0800 (PST) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id JAA07386 for ; Tue, 17 Dec 1996 09:17:50 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id KAA06227; Tue, 17 Dec 1996 10:17:38 -0700 Message-Id: <199612171717.KAA06227@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: Arlie Davis cc: "FreeBSD SMP List (E-mail)" , Michael Jung Subject: Re: As of 10AM, Dec 17, SMP is pretty solid. In-reply-to: Your message of "Tue, 17 Dec 1996 10:50:18 EST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 17 Dec 1996 10:17:38 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, >Just wanted to say THANKS for the good work! The kernel is solid; we've >been beating it to death since we turned on the second processor, and it >hasn't flinched. We're running without the APIC_IO option, which I >understand is more advanced but currently more risky, so this is hardly >new news. We are building a kernel with APIC_IO, and will report on >that, later. APIC_IO on the P5s seems solid, its just the P6s that have a problem. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Tue Dec 17 09:24:59 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id JAA07941 for smp-outgoing; Tue, 17 Dec 1996 09:24:59 -0800 (PST) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id JAA07931 for ; Tue, 17 Dec 1996 09:24:54 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id KAA06286; Tue, 17 Dec 1996 10:24:38 -0700 Message-Id: <199612171724.KAA06286@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: "Bror 'Count' Heinola" cc: freebsd-smp@freebsd.org Subject: Re: Success! ...well, not really In-reply-to: Your message of "Tue, 17 Dec 1996 16:06:12 +0200." <199612171406.QAA05468@key.sms.fi> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 17 Dec 1996 10:24:38 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, > I'm a proud owner of a Tyan Titan Pro S1668 ATX dual Ppro > motherboard with two PPro 180MHz CPU's. > ... > The machine runs just fine with both kernels when the second > cpu is disabled, however, when I enable the second CPU it > works, but whenever I try to "make" something, it immediately > bombs with a panic like this (btw, is there a way of saving a > panic output in short of writing it down by hand?) : > > Fatal trap 12: page fault while in kernel mode > cpunumber = 0 > fault virtual address = 0xfffbeff4 > fault code = supervisor read, page not present > instruction pointer = 0x8:0xf0176c13 > stack pointer = 0x10:0xefbffe94 > frame pointer = 0x10:0xefbffeb0 > code segment = base 0x0, limit 0xfffff, type ox1b > = dpl 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 190 (sh) > interrupt mask = > trap number = 12 > panic (cpu#0): page fault > boot() called on cpu#0 >... > io0 (APIC): apic id: 2, version: 0x00170011 >... > Enabled INTs: 1, 2, 3, 4, 6, 7, 8, 10, 12, 16, imen: 0x00feea21 > SMP: All idle procs online. > SMP: Starting 1st AP! > SMP: AP CPU #1 LAUNCHED!! Starting Scheduling... > SMP: TADA! CPU #1 made it into the scheduler!. > SMP: All 2 CPU's are online! >... > options SMP # Symmetric MultiProcessor Kernel > options APIC_IO # Symmetric (APIC) I/O > options NCPU=2 # number of CPUs > options NBUS=2 # number of busses > options NAPIC=1 # number of IO APICs > options NINTR=24 # number of INTs this is a known problem with the P6 machines. add: options SMP_INVLTLB and see if it helps (probably wont, but we need the feedback) The solution is rather complex, we're discussing an attack at the moment... Follow the discussion on "TLB shootdowns" for details. -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Tue Dec 17 15:51:17 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id PAA00224 for smp-outgoing; Tue, 17 Dec 1996 15:51:17 -0800 (PST) Received: from extractor.compass-da.com (firewall-user@gateway.compass-da.com [162.17.253.11]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id PAA00208 for ; Tue, 17 Dec 1996 15:51:13 -0800 (PST) Received: by extractor.compass-da.com; id AA28634; Tue, 17 Dec 96 15:51:04 PST Received: from luminary.sanjose.compass-da.com(162.17.1.44) by gateway.compass-da.com via smap (3.2) id xma028626; Tue, 17 Dec 96 15:50:44 -0800 Received: from mirage.compass-da.com (mirage [162.17.2.39]) by luminary.sanjose.compass-da.com (8.7.1/8.7.1) with SMTP id PAA08006; Tue, 17 Dec 1996 15:50:49 -0800 (PST) Received: from localhost by mirage.compass-da.com (SMI-8.6/SMI-SVR4) id PAA24994; Tue, 17 Dec 1996 15:55:06 -0800 Date: Tue, 17 Dec 1996 15:55:06 -0800 (PST) From: Garth Corral X-Sender: garthc@mirage To: Steve Passe Cc: freebsd-smp@freebsd.org Subject: Re: AMI Merlin DP Motherboard In-Reply-To: <199612170549.WAA02970@clem.systemsix.com> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Mon, 16 Dec 1996, Steve Passe wrote: > Hi, > > > > > I'm about to take a plung and buy the AMI Merlin DP Dual Pentium Pro > > motherboard (my second choice is the SuperMicro P6DOF). Has anyone tired > > running the SMP kernel on this puppy yet? Is there any reason I should > > not buy this MB? I know the SMP kernel is not very stable on P6 yet, and > > I know it will take several more month before it happens. > > I haven't received any reports on this board. In general AMI makes good > hardware. Is it an EISA motherboard? I would caution against it if so. > In general EISA boards can give grief under the SMP spec even if you > don't use EISA cards. Unfortunately the spec allows some pretty braindead > behaviour for EISA systems. Some are fine, it depends on the chipset > and design used by the MB manufacturer. > Would you caution against the purchase of any MP EISA board or just P6? I have several EISA cards from my current system that I would like to reuse and am considering the purchase of an AMI Titan III motherboard. The cost of replacing these with PCI cards makes the cost of upgrading prohibitive. Do these motherboards cause the same problems with other (Sun, SCO) SMP implementations? Does anyone have any comments either negative or positive about this particular motherboard? Many thanks in advance, > -- > Steve Passe | powered by > smp@csn.net | FreeBSD > Garth Corral From owner-freebsd-smp Tue Dec 17 16:42:42 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id QAA04556 for smp-outgoing; Tue, 17 Dec 1996 16:42:42 -0800 (PST) Received: from cs.utah.edu (cs.utah.edu [128.110.4.21]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id QAA04551 for ; Tue, 17 Dec 1996 16:42:38 -0800 (PST) Received: from fast.cs.utah.edu by cs.utah.edu (8.8.4/utah-2.21-cs) id RAA21543; Tue, 17 Dec 1996 17:42:29 -0700 (MST) Received: by fast.cs.utah.edu (8.6.10/utah-2.15-leaf) id RAA26079; Tue, 17 Dec 1996 17:42:27 -0700 Date: Tue, 17 Dec 1996 17:42:27 -0700 From: vanmaren@fast.cs.utah.edu (Kevin Van Maren) Message-Id: <199612180042.RAA26079@fast.cs.utah.edu> To: garthc@compass-da.com, smp@csn.net Subject: Re: AMI Merlin DP Motherboard Cc: freebsd-smp@freebsd.org Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >Would you caution against the purchase of any MP EISA board or just P6? I >have several EISA cards from my current system that I would like to reuse >and am considering the purchase of an AMI Titan III motherboard. The cost >of replacing these with PCI cards makes the cost of upgrading prohibitive. >Do these motherboards cause the same problems with other (Sun, SCO) SMP >implementations? Does anyone have any comments either negative or >positive about this particular motherboard? Rest assured, EISA support will be added, and FreeBSD WILL have to deal with the stupid things people can do. I think Steve was just saying (feel free to correct me) that EISA support isn't going to happen Really Soon, and an EISA motherboard will therefore have difficulties under the current FreeBSD SMP (even w/o EISA cards). I would buy EISA. (Then you can help debug it!) From owner-freebsd-smp Tue Dec 17 17:11:42 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id RAA05723 for smp-outgoing; Tue, 17 Dec 1996 17:11:42 -0800 (PST) Received: from avatar.avatar.com (avatar.avatar.com [199.33.206.17]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id RAA05705 for ; Tue, 17 Dec 1996 17:11:40 -0800 (PST) Received: from avatar.avatar.com (kory@avatar.avatar.com [199.33.206.17]) by avatar.avatar.com (8.7.4/8.6.9) with SMTP id RAA10339; Tue, 17 Dec 1996 17:10:06 -0800 (PST) Date: Tue, 17 Dec 1996 17:10:03 -0800 (PST) From: Kory Hamzeh To: Kevin Van Maren cc: garthc@compass-da.com, smp@csn.net, freebsd-smp@freebsd.org Subject: Re: AMI Merlin DP Motherboard In-Reply-To: <199612180042.RAA26079@fast.cs.utah.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Unless I'm totaly confused (very likely!), the AMI Merlin DP is PCI based. Can someone explain why the P5 works by P6 does not? Kory On Tue, 17 Dec 1996, Kevin Van Maren wrote: > >Would you caution against the purchase of any MP EISA board or just P6? I > >have several EISA cards from my current system that I would like to reuse > >and am considering the purchase of an AMI Titan III motherboard. The cost > >of replacing these with PCI cards makes the cost of upgrading prohibitive. > >Do these motherboards cause the same problems with other (Sun, SCO) SMP > >implementations? Does anyone have any comments either negative or > >positive about this particular motherboard? > > Rest assured, EISA support will be added, and FreeBSD WILL > have to deal with the stupid things people can do. I think > Steve was just saying (feel free to correct me) that EISA > support isn't going to happen Really Soon, and an EISA > motherboard will therefore have difficulties under the current > FreeBSD SMP (even w/o EISA cards). > > I would buy EISA. (Then you can help debug it!) > From owner-freebsd-smp Tue Dec 17 17:51:34 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id RAA07887 for smp-outgoing; Tue, 17 Dec 1996 17:51:34 -0800 (PST) Received: from ilsa.systemsix.com (ilsa.systemsix.com [198.99.86.129]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id RAA07882 for ; Tue, 17 Dec 1996 17:51:29 -0800 (PST) Received: (from smp@localhost) by ilsa.systemsix.com (8.7.5/8.6.12) id SAA15012 for freebsd-smp@freebsd.org; Tue, 17 Dec 1996 18:51:21 -0700 (MST) Date: Tue, 17 Dec 1996 18:51:21 -0700 (MST) From: Steve Passe Message-Id: <199612180151.SAA15012@ilsa.systemsix.com> To: freebsd-smp@freebsd.org Subject: Re: AMI Merlin DP Motherboard Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >Would you caution against the purchase of any MP EISA board or just P6? any, the EISA issues are unrelated to P5/P6 stuff (kinda, this is complex) Some EISA systems do not connect the system clock (8254) to the IO APIC.This requires "mixed-mode" programming the IO APIC AND the 8259 ICU. I have already written the code to do this and it appears to work. BUT this is a small in-efficiency for hard-clock INTs. So far ALL EISA systems I have seen use the ISA INTs 0-15 redirection capability for routing both EISA AND PCI INTs to the lower 16 ISA INT vectors. some (majority, but NOT all) PCI/ISA systems route the PCI INTs to upper (ie above 16) IO APIC INTs, thus giving you 4 or more extra INT vectors, helping greatly with the typical "not enough INT choices" on the ISA bus. this fact is the major reason I have for avoiding EISA boards. you need to go thru the mptable database to see details about what I am talking about here. >Do these motherboards cause the same problems with other (Sun, SCO) SMP implementations? The issue is "Intel MP spec motherboards". this means among other things ONLY Intel CPU's (P5/P6). In otherwords its strictly a hardware issue, NOT an OS issue. From owner-freebsd-smp Tue Dec 17 18:02:54 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id SAA08547 for smp-outgoing; Tue, 17 Dec 1996 18:02:54 -0800 (PST) Received: from ilsa.systemsix.com (ilsa.systemsix.com [198.99.86.129]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id SAA08536 for ; Tue, 17 Dec 1996 18:02:49 -0800 (PST) Received: (from smp@localhost) by ilsa.systemsix.com (8.7.5/8.6.12) id TAA15086 for freebsd-smp@freebsd.org; Tue, 17 Dec 1996 19:02:39 -0700 (MST) Date: Tue, 17 Dec 1996 19:02:39 -0700 (MST) From: Steve Passe Message-Id: <199612180202.TAA15086@ilsa.systemsix.com> To: freebsd-smp@freebsd.org Subject: Re: AMI Merlin DP Motherboard Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >Unless I'm totaly confused (very likely!), the AMI Merlin DP is PCI based. it could be both PCI and EISA, ie PCI/EISA, whil Im suggesting it is desirable to get PCI/ISA. Im not trying to start a religious war here, just pointing out the differences. If you have NO EISA crds why put up with the limitations (primarily 4 less INT sources) of the EISA bus. If you do have EISA cards you want to re-use, and don't have an INT crunch, then EISA should work fine. Note that I have seen at least 1 PCI/ISA board that still requires "mixwd-mode" even though not an EISA board. And not all PCI/ISA boards take advantage of the upper APIC INTS (16-23). you really need to do your homework here. >Can someone explain why the P5 works by P6 does not? If we knew we could probably fix it!!! we have theories BUT we don't know for sure... search the mail list for "TLB shootdown" for the discussions. From owner-freebsd-smp Tue Dec 17 18:39:02 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id SAA11064 for smp-outgoing; Tue, 17 Dec 1996 18:39:02 -0800 (PST) Received: from cs.utah.edu (cs.utah.edu [128.110.4.21]) by freefall.freebsd.org (8.8.4/8.8.4) with ESMTP id SAA11051 for ; Tue, 17 Dec 1996 18:38:58 -0800 (PST) Received: from fast.cs.utah.edu by cs.utah.edu (8.8.4/utah-2.21-cs) id TAA22836; Tue, 17 Dec 1996 19:38:50 -0700 (MST) Received: by fast.cs.utah.edu (8.6.10/utah-2.15-leaf) id TAA00379; Tue, 17 Dec 1996 19:38:49 -0700 Date: Tue, 17 Dec 1996 19:38:49 -0700 From: vanmaren@fast.cs.utah.edu (Kevin Van Maren) Message-Id: <199612180238.TAA00379@fast.cs.utah.edu> To: freebsd-smp@freebsd.org, smp@ilsa.systemsix.com Subject: Re: AMI Merlin DP Motherboard Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk >>Do these motherboards cause the same problems with other (Sun, SCO) SMP >>implementations? >The issue is "Intel MP spec motherboards". this means among other things >ONLY Intel CPU's (P5/P6). In otherwords its strictly a hardware issue, NOT >an OS issue. I think the question was does Solaris for the x86 have problems with EISA motherboards. I know NT works fine; I assume Solaris does as well. From owner-freebsd-smp Wed Dec 18 10:10:53 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id KAA19784 for smp-outgoing; Wed, 18 Dec 1996 10:10:53 -0800 (PST) Received: from extractor.compass-da.com (firewall-user@gateway.compass-da.com [162.17.253.11]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id KAA19777 for ; Wed, 18 Dec 1996 10:10:49 -0800 (PST) Received: by extractor.compass-da.com; id AA29932; Wed, 18 Dec 96 10:10:34 PST Received: from luminary.sanjose.compass-da.com(162.17.1.44) by gateway.compass-da.com via smap (3.2) id xma029920; Wed, 18 Dec 96 10:10:25 -0800 Received: from mirage.compass-da.com (mirage [162.17.2.39]) by luminary.sanjose.compass-da.com (8.7.1/8.7.1) with SMTP id KAA27882; Wed, 18 Dec 1996 10:10:30 -0800 (PST) Received: from localhost by mirage.compass-da.com (SMI-8.6/SMI-SVR4) id KAA25618; Wed, 18 Dec 1996 10:14:50 -0800 Date: Wed, 18 Dec 1996 10:14:50 -0800 (PST) From: Garth Corral X-Sender: garthc@mirage To: Kevin Van Maren Cc: freebsd-smp@freebsd.org, smp@ilsa.systemsix.com Subject: Re: AMI Merlin DP Motherboard In-Reply-To: <199612180238.TAA00379@fast.cs.utah.edu> Message-Id: Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On Tue, 17 Dec 1996, Kevin Van Maren wrote: > I think the question was does Solaris for the x86 have problems > with EISA motherboards. I know NT works fine; I assume > Solaris does as well. > Yes, this is it exactly. Sorry if I wasn't clear. I was wondering whether these architectural deficiencies had been addressed in these other SMP implementations and, if so, what limitations result. Thanks for helping me clarify this. Garth From owner-freebsd-smp Wed Dec 18 19:10:05 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id TAA25886 for smp-outgoing; Wed, 18 Dec 1996 19:10:05 -0800 (PST) Received: from mail001.mediacity.com (mail001.mediacity.com [206.24.105.68]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id TAA25881 for ; Wed, 18 Dec 1996 19:10:03 -0800 (PST) From: brian@mediacity.com Received: (qmail-queue invoked from smtpd); 19 Dec 1996 03:09:46 -0000 Received: from home001.mediacity.com (HELO mediacity.com) (qmailr@206.24.105.66) by mail001.mediacity.com with SMTP; 19 Dec 1996 03:09:46 -0000 Received: (qmail-queue invoked by uid 100); 19 Dec 1996 03:09:03 -0000 Message-ID: <19961219030903.23357.qmail@mediacity.com> Subject: A successful launch To: freebsd-smp@freebsd.org Date: Wed, 18 Dec 1996 19:09:03 -0800 (PST) Reply-To: brian@mediacity.com X-Mailer: ELM [version 2.4ME+ PL22 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Following the instructions from 'Installing the FreeBSD SMP Kernel' I just brought up the SMP kernel on an ASUS motherboard with 2xPP200 w/128MB per processor. FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 1, version: 0x00040011 cpu1 (AP): apic id: 0, version: 0x00040011 Warning: APIC I/O disabled ... SMP: All idle procs online. SMP: Starting 1st AP! SMP: AP CPU #1 LAUNCHED!! Starting Scheduling... SMP: TADA! CPU #1 made it into the scheduler!. SMP: All 2 CPU's are online! Shall I enable the APIC_IO code or will bad things happen? I had no troubles following the docs. BTW, what do I do to demonstrate both processers are doing things? -- Brian Litzinger Powered by FreeBSD http[s]://www.mpress.com From owner-freebsd-smp Wed Dec 18 21:51:41 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id VAA01931 for smp-outgoing; Wed, 18 Dec 1996 21:51:41 -0800 (PST) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id VAA01924 for ; Wed, 18 Dec 1996 21:51:38 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id WAA16302; Wed, 18 Dec 1996 22:51:28 -0700 Message-Id: <199612190551.WAA16302@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: brian@mediacity.com cc: freebsd-smp@freebsd.org Subject: Re: A successful launch In-reply-to: Your message of "Wed, 18 Dec 1996 19:09:03 PST." <19961219030903.23357.qmail@mediacity.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 18 Dec 1996 22:51:28 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, > Following the instructions from 'Installing the FreeBSD SMP Kernel' > > I just brought up the SMP kernel on an ASUS motherboard with > 2xPP200 w/128MB per processor. > > FreeBSD/SMP: Multiprocessor motherboard > cpu0 (BSP): apic id: 1, version: 0x00040011 > cpu1 (AP): apic id: 0, version: 0x00040011 > Warning: APIC I/O disabled > ... > SMP: All idle procs online. > SMP: Starting 1st AP! > SMP: AP CPU #1 LAUNCHED!! Starting Scheduling... > SMP: TADA! CPU #1 made it into the scheduler!. > SMP: All 2 CPU's are online! > > Shall I enable the APIC_IO code or will bad things happen? not a good idea right now... more likely to tickle the P6 bug. keep track of any problems for a few days, then perhaps try it to see if things become less stable. ` -- Steve Passe | powered by smp@csn.net | FreeBSD From owner-freebsd-smp Thu Dec 19 22:52:51 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id WAA08035 for smp-outgoing; Thu, 19 Dec 1996 22:52:51 -0800 (PST) Received: from mail001.mediacity.com (mail001.mediacity.com [206.24.105.68]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id WAA08030 for ; Thu, 19 Dec 1996 22:52:49 -0800 (PST) From: brian@mediacity.com Received: (qmail-queue invoked from smtpd); 20 Dec 1996 06:52:35 -0000 Received: from home001.mediacity.com (HELO mediacity.com) (qmailr@206.24.105.66) by mail001.mediacity.com with SMTP; 20 Dec 1996 06:52:35 -0000 Received: (qmail-queue invoked by uid 100); 20 Dec 1996 06:51:47 -0000 Message-ID: <19961220065147.28067.qmail@mediacity.com> Subject: P6 panics at same point To: freebsd-smp@freebsd.org Date: Thu, 19 Dec 1996 22:51:47 -0800 (PST) Reply-To: brian@mediacity.com X-Mailer: ELM [version 2.4ME+ PL22 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk On a ASUS P65UP5 2xPP200 MB with 256MB RAM I can start the second processory, but if I try to build apache from the ports, or most anything else complex I get: Fatal Trap 12: page fault while in kernel mode cpunumber=0 fault virtual address = 0xffc00148 fault code = supervisor read, page not present instruction pointer = 0x8:0xf01c0757 stack pointer = 0x10:0xefbffe94 frame pointer = 0x10:0xefbffeb0 code segment = base 0x0, limit 0xfffff, type 0x1b processor eflags = DPL 0, pres 1, def 32, gran 1 = interrupt enabled, resume, IOPL=0 current process = 185 (sh) interrupt mask = kernel: type 12 trap, code=0 stopped at _pmap_enter+0x85: movl 0(%ecx),%ecx It happens basically the same each time it happens. If I run the SMP kernel without starting the 2nd processor all goes well. -- Brian Litzinger Powered by FreeBSD http[s]://www.mpress.com From owner-freebsd-smp Fri Dec 20 12:07:31 1996 Return-Path: Received: (from root@localhost) by freefall.freebsd.org (8.8.4/8.8.4) id MAA06150 for smp-outgoing; Fri, 20 Dec 1996 12:07:31 -0800 (PST) Received: from clem.systemsix.com (clem.systemsix.com [198.99.86.131]) by freefall.freebsd.org (8.8.4/8.8.4) with SMTP id MAA06145 for ; Fri, 20 Dec 1996 12:07:25 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by clem.systemsix.com (8.6.12/8.6.12) with SMTP id NAA26381; Fri, 20 Dec 1996 13:06:57 -0700 Message-Id: <199612202006.NAA26381@clem.systemsix.com> X-Authentication-Warning: clem.systemsix.com: Host localhost didn't use HELO protocol X-Mailer: exmh version 1.6.5 12/11/95 From: Steve Passe To: brian@mediacity.com cc: freebsd-smp@freebsd.org Subject: Re: P6 panics at same point In-reply-to: Your message of "Thu, 19 Dec 1996 22:51:47 PST." <19961220065147.28067.qmail@mediacity.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 20 Dec 1996 13:06:56 -0700 Sender: owner-smp@freebsd.org X-Loop: FreeBSD.org Precedence: bulk Hi, > On a ASUS P65UP5 2xPP200 MB with 256MB RAM I can start the > second processory, but if I try to build apache from the ports, > or most anything else complex I get: > > Fatal Trap 12: page fault while in kernel mode >... > stopped at _pmap_enter+0x85: movl 0(%ecx),%ecx > > It happens basically the same each time it happens. > > If I run the SMP kernel without starting the 2nd processor all > goes well. this is THE known current problem, affects everyone with P6 machines, but doesn't occur on P5 machines. Since Peter and I both have P5 machines we haven't been able to directly address it. We gotta do something soon, but I'm not sure what. I'm up to my eyeballs right now in 'real work', so I haven't even been able to ponder it much. Wish I could give you a better answer. Can anyone give a definitive summary of the problem and what we currently know(think we know) about it? -- Steve Passe | powered by smp@csn.net | FreeBSD