From owner-freebsd-smp Mon Jan 1 4:18:36 2001 From owner-freebsd-smp@FreeBSD.ORG Mon Jan 1 04:18:34 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from silver.he.net (silver.he.net [216.218.151.2]) by hub.freebsd.org (Postfix) with ESMTP id AE30837B400 for ; Mon, 1 Jan 2001 04:18:34 -0800 (PST) Received: from calsoftinc.com ([208.149.99.4] (may be forged)) by silver.he.net (8.8.6/8.8.2) with ESMTP id EAA11124 for ; Mon, 1 Jan 2001 04:18:30 -0800 Sender: asr@silver.he.net Message-ID: <3A507780.54341D88@calsoftinc.com> Date: Mon, 01 Jan 2001 17:56:40 +0530 From: "Ashutosh S. Rajekar Asr" X-Mailer: Mozilla 4.72 [en] (X11; I; Linux 2.2.12 i386) X-Accept-Language: en MIME-Version: 1.0 To: freebsd-smp@freebsd.org Subject: FreeBSD SMP support Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hello, I was just wondering about how far the SMP support has reached in FreeBSD. According to my impression, I feel that FreeBSD has moved away from the single "GIANT" SMP lock, and uses separate locks for subsystems like the ISR routines, spl system, etc. This should get preformance improvements compared to Linux 2.2, which used a single kernel lock, effectively permitting only one thread (or process) to be in the kernel at the same time. Please correct me if I am wrong, and preferably provide some pointer to documents pertaining to the same. Thanks, Ashutosh S. Rajekar To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Jan 1 11:24:35 2001 From owner-freebsd-smp@FreeBSD.ORG Mon Jan 1 11:24:33 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from peorth.iteration.net (peorth.iteration.net [208.190.180.178]) by hub.freebsd.org (Postfix) with ESMTP id 4844B37B400 for ; Mon, 1 Jan 2001 11:24:33 -0800 (PST) Received: by peorth.iteration.net (Postfix, from userid 1001) id 3515A574D5; Mon, 1 Jan 2001 13:25:08 -0600 (CST) Date: Mon, 1 Jan 2001 13:25:08 -0600 From: "Michael C . Wu" To: "Ashutosh S. Rajekar Asr" Cc: freebsd-smp@freebsd.org Subject: Re: FreeBSD SMP support Message-ID: <20010101132508.A51028@peorth.iteration.net> Reply-To: "Michael C . Wu" Mail-Followup-To: "Michael C . Wu" , "Ashutosh S. Rajekar Asr" , freebsd-smp@freebsd.org References: <3A507780.54341D88@calsoftinc.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3A507780.54341D88@calsoftinc.com>; from asr@calsoftinc.com on Mon, Jan 01, 2001 at 05:56:40PM +0530 X-PGP-Fingerprint: 5025 F691 F943 8128 48A8 5025 77CE 29C5 8FA1 2E20 X-PGP-Key-ID: 0x8FA12E20 Sender: keichii@peorth.iteration.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Mon, Jan 01, 2001 at 05:56:40PM +0530, Ashutosh S. Rajekar Asr scribbled: | I was just wondering about how far the SMP support has reached in | FreeBSD. According to my impression, I feel that FreeBSD has moved away | from the single "GIANT" SMP lock, and uses separate locks for subsystems | like the ISR routines, spl system, etc. This should get preformance | improvements compared to Linux 2.2, which used a single kernel lock, | effectively permitting only one thread (or process) to be in the kernel | at the same time. | | Please correct me if I am wrong, and preferably provide some pointer to | documents pertaining to the same. /usr/src, CVSweb, and -SMP archives. :) -- +------------------------------------------------------------------+ | keichii@peorth.iteration.net | keichii@bsdconspiracy.net | | http://peorth.iteration.net/~keichii | Yes, BSD is a conspiracy. | +------------------------------------------------------------------+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Jan 1 13:48:20 2001 From owner-freebsd-smp@FreeBSD.ORG Mon Jan 1 13:48:18 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from assaris.sics.se (dyna225-140.nada.kth.se [130.237.225.140]) by hub.freebsd.org (Postfix) with ESMTP id 1D91C37B400 for ; Mon, 1 Jan 2001 13:48:18 -0800 (PST) Received: (from assar@localhost) by assaris.sics.se (8.9.3/8.9.3) id WAA18506; Mon, 1 Jan 2001 22:48:10 +0100 (CET) (envelope-from assar) Sender: assar@assaris.sics.se To: "Josep M. Blanquer" Cc: Matt Dillon , smp@FreeBSD.ORG Subject: Re: getting processor's cacheline size? References: <200012290623.eBT6NaJ03589@mass.osd.bsdi.com> <200012290636.eBT6a2m29328@earth.backplane.com> <20001229015117.F8848@cs.rice.edu> <200012291806.eBTI6jL32222@earth.backplane.com> <12D3BADB.3E1BB0BA@cs.ucsb.edu> From: Assar Westerlund Date: 01 Jan 2001 22:48:10 +0100 In-Reply-To: "Josep M. Blanquer"'s message of "Fri, 04 Jan 1980 21:30:19 +0100" Message-ID: <5l3df37xlx.fsf@assaris.sics.se> Lines: 9 User-Agent: Gnus/5.070098 (Pterodactyl Gnus v0.98) Emacs/20.6 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org "Josep M. Blanquer" writes: > I´m putting something toghether at http://steamboat.cs.ucsb.edu/mob > but not all links are up yet. Not much information is available by now > (a paper is submitted) but if anybody wants to give it a try feel free > to contact me. Thanks, now added as ports/devel/mob. /assar To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Jan 1 20: 2:51 2001 From owner-freebsd-smp@FreeBSD.ORG Mon Jan 1 20:02:48 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fast.cs.utah.edu (fast.cs.utah.edu [155.99.212.1]) by hub.freebsd.org (Postfix) with ESMTP id 98FBE37B400; Mon, 1 Jan 2001 20:02:47 -0800 (PST) Received: (from vanmaren@localhost) by fast.cs.utah.edu (8.9.1/8.9.1) id VAA12154; Mon, 1 Jan 2001 21:02:46 -0700 (MST) Date: Mon, 1 Jan 2001 21:02:46 -0700 (MST) From: Kevin Van Maren Message-Id: <200101020402.VAA12154@fast.cs.utah.edu> To: jhb@FreeBSD.ORG, julian@elischer.org Subject: Re: looking for locking advice.. Cc: archie@FreeBSD.ORG, smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Reader/Writer Locks (or SIX Locks): Julian, I downloaded your code yesterday from the web. I really like your reader/writer queue implementation. You basically use mutexes to protect a "busy flag" and a jobs queue, and support running several "read-only" jobs at once. Your implementation is truly "fair" (keeps requests in order). You don't hold the mutexes for long, so there should be little chance of a thread blocking on the mutex. An adaptive (spin-then- sleep) mutex should be perfect to build them on (I know you want them parallel to mtx, not built on them). There was also your comment: "I have also for data going through...on failure, simply queue... never block". Since you must acquire the mutex that protects the queue, I assume you mean you won't block anywhere else, and are describing the model I mention above? This is very similar to what I've been trying to get added to the UDI execution environment. Maybe in 1.02. [www.project-udi.org]** While not exactly "problems", I think you should explicitly mention the behavior patterns of your implementation: When there is a pending/active writer, the queue provides "fan-in" (put all requests on a single Q), but there is no way to "fan out" the queue to a pool of "reader threads" when the writer completes. Almost like you need a condition variable for idle reader threads to sleep on, which you signal whenever a writer completes (all while avoiding N CPUs pounding the mutex at exactly the same time...). But this probably isn't a problem: eventually the queue will empty, and it will empty read-only requests faster if more requests come in (each with another context to run items). So it will just start emptying them perhaps a little slower than it could, but saves a lot of overhead and complexity. If there are a lot of reader threads trying to run, a writer thread could possibly "livelock" before it gets to execute because a fastrack thread can break the condition for the writer thread to execute. Yes, the reader thread will then dequeue, but a new reader thread could break the condition for *that* thread. [Atomic-compare-and-swap could be used in acquire_read, but that would livelock the readers instead of a possible writer. More on atomic ops in a following email.] I also think a reader/writer implementation should provide an API to "upgrade" a reader lock to a writer lock, and to "downgrade" a writer lock to a reader lock. While I don't expect upgrades/ downgrades to occur often, they are sometimes necessary. For example, if I process incomming packets and pass them on, but I queue out-of-order packets internally, I need to upgrade to a writer lock to append the packet to my Q. I don't like the "hack" of resending it to myself marked as a writer... Since it isn't always possible to know in advance if writer permissions are required, make it easy on people. Looking over your code, there were a few things that bothered me a little. The first was your not using the STAILQ macros, so I re-wrote it to use them instead of your manually-inlined expansions. Even if you save an instruction, the macros are SO much easier to read... I may have changed the behavior slightly (I went more by your comments than the pointer-munging code). STAILQ macros are properly protected the the mutex, just like your code. I also re-wrote the code to make "cpp" do the math: instead of relying on a binary carry to set the correct bit, I add the correct value and subtract the old one. Code gen is the same, but the C source is more readable and doesn't require the special bit placements. Atomic "set" and "clear" operations could also be used, but then it might require two operations (second to increment the reader count). Since I got "flag" and "flags" mixed up once, I renamed them to make it clear which was which (q_flags, el_flags; still not happy with the names). Fixed a couple spelling typos while I was at it. I noticed that "dummy" assumed that inputs were always read-only. Modified it to check and acquire writer perms if they were needed (since writers come in *somewhere*; I know it is just example code, but I wanted it to be more "real"). Once I did all that, it was much easier for me to "see" what was going on with the code. I think I've found a race condition in the code that can lead to a writer AND a reader being active at the same time: the problem is the queue flags are both manipulated by (multiple) atomic operations and protected by the mutex. It appears possible that while a thread is in acquire_write(), after the check ((ngq->q_flags & ~SINGLE_THREAD_ONLY) == 0) is true, but before WRITER_ACTIVE is added to q_flags, a fastrack reader can come in, atomically add READER_INCREMENT, and verify that there isn't a writer active. Then, the writer becomes active and executes. I'm sending my modified version to Julian off-line. He can take as many of my changes as he wants. Email him or me if you'd like a copy. Kevin ** [I work on UDI drivers (and some on the spec) during the day. UDI's execution environment is a little "strange", but consists of only a single non-blocking request active in a "region" at a time. I want to extend it to allow multiple "read only" requests or a single writer request. A "read-only" request in UDI probably looks different from in netgraph or FreeBSD, though. One problem with RW locks in UDI is that there is typically a single "writer" operation required: add or remove an element from a queue, which is protected by the execution environment.] To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Jan 1 21:10:14 2001 From owner-freebsd-smp@FreeBSD.ORG Mon Jan 1 21:10:12 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fast.cs.utah.edu (fast.cs.utah.edu [155.99.212.1]) by hub.freebsd.org (Postfix) with ESMTP id 9084537B400 for ; Mon, 1 Jan 2001 21:10:11 -0800 (PST) Received: (from vanmaren@localhost) by fast.cs.utah.edu (8.9.1/8.9.1) id WAA13199; Mon, 1 Jan 2001 22:10:05 -0700 (MST) Date: Mon, 1 Jan 2001 22:10:05 -0700 (MST) From: Kevin Van Maren Message-Id: <200101020510.WAA13199@fast.cs.utah.edu> To: cp@bsdi.com, smp@FreeBSD.ORG Subject: Re: atomic increment? Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I didn't see the Jason Evans flame thread on -arch. Does anyone have a pointer to it in the mail archive? In the interim, I think atomic_{increment,decrement}, even if they are just syntactic sugar to atomic_{add,subtract}, should be provided. After all, we use "++" as syntactic sugar to "+=1". [The fact that gcc uses an intermediate register to add an immediate constant is bogus, and not sufficient reason by itself to use atomic_increment.] On x86 the code is slightly better for "++" ver "+=1", while it really is just syntactic sugar on other systems. However, I also have another thought. Often times I need to modify a value and also (atomically) determine it's (old|new) value. Primitives that use "xadd" instead of "add" or "sub" provide atomicity and eliminate an extra read. Yes, trashing the added value may cause a register to spill if the value is needed again later, but often times it is never used again anyway. [pre-processor can negate the "sub"; worst case we need an extra "neg" instruction for atomic_subtract, but the subtraction will still be atomic.] Essentially, the (old) value is available for free, so why not provide it? It might even make sense to always provide the old value for add/subtract, and have gcc throw away the unused output (unless the input value is reused, when we'd lose a whole register to the xadd). Even if the processor does not support xadd-like operations, it can be emulated (more expensively) using load, add, cmpxchg, loop- if-failed [similar to the code frag below, but with a while() loop.] But by providing a primitive, it can be optimized much further than just C code using an atomic cmpxchg operation (and is "negative cost" on x86 -- faster than the non-atomic version). For example, in Julian's acquire_reader: => atomic_add_long(&ngq->q_flags, READER_INCREMENT); => if ((ngq->q_flags & (~READER_MASK)) == 0) { can be changed to (find a better macro name): => flags = atomic_read_add_long(&ngq->q_flags, READER_INCREMENT); => if ((flags & (~READER_MASK)) == 0) { gcc will realize that the register is sticks READER_INCREMENT in (which it shouldn't for normal atomic_adds, but does anyway) now contains "flags", and will use that register again for the mask, thus saving a memory access to "volatile" memory. In Julian's acquire_writer, we need to do an atomic compare-and-swap operation, instead of assuming two operations are atomic (because the above acquire_reader code could be executed between the two following statements): => if ((ngq->q_flags & (~SINGLE_THREAD_ONLY)) == 0) { => atomic_add_long(&ngq->q_flags, WRITER_ACTIVE); Here is a possible code sequence to "just get it working" (at least I *think* this fixes the alleged problem): [ register int flags; ] => flags = ngq->q_flags; => if ((flags & (~SINGLE_THREAD_ONLY) == 0) && => atomic_cmpset(&ngq->q_flags, flags, flags + WRITER_ACTIVE)) { One more thought on atomic operations: If we don't assume assignments are atomic, and always use atomic_load and atomic_store, then we a) can easily provide atomic 64-bit operations on x86 (quick hack would be to use a single mutex for all 64-bit operations), and b) we can port to platforms where atomic_add requires a mutex to protect the atomic_add or atomic_cmpset sequence. [Slow as molasses] On x86, the load/store macros are NOPs, but the use also (c) makes it clear that we are manipulating a variable we perform atomic operations on. Kevin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 11:36:52 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 11:36:46 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from meow.osd.bsdi.com (meow.osd.bsdi.com [204.216.28.88]) by hub.freebsd.org (Postfix) with ESMTP id E2E5437B715 for ; Tue, 2 Jan 2001 11:36:43 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by meow.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id f02JZtG01040; Tue, 2 Jan 2001 11:35:55 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200101020510.WAA13199@fast.cs.utah.edu> Date: Tue, 02 Jan 2001 11:36:19 -0800 (PST) From: John Baldwin To: Kevin Van Maren Subject: Re: atomic increment? Cc: smp@FreeBSD.org, cp@bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 02-Jan-01 Kevin Van Maren wrote: > I didn't see the Jason Evans flame thread on -arch. Does > anyone have a pointer to it in the mail archive? > > In the interim, I think atomic_{increment,decrement}, even if > they are just syntactic sugar to atomic_{add,subtract}, should be > provided. After all, we use "++" as syntactic sugar to "+=1". > [The fact that gcc uses an intermediate register to add an immediate > constant is bogus, and not sufficient reason by itself to use > atomic_increment.] On x86 the code is slightly better for "++" > ver "+=1", while it really is just syntactic sugar on other systems. There is also a desire to try and keep the atomic API from being too huge I think. Atomic operations are _expensive_. One thing you are forgetting about on the x86 is that an atomic op on an SMP system requires a 'lock' prefix. The cost of locking the bus drowns out the savings you may get by getting one or two less instructions. > However, I also have another thought. Often times I need to > modify a value and also (atomically) determine it's (old|new) > value. Primitives that use "xadd" instead of "add" or "sub" > provide atomicity and eliminate an extra read. Yes, trashing > the added value may cause a register to spill if the value is > needed again later, but often times it is never used again anyway. > [pre-processor can negate the "sub"; worst case we need an extra > "neg" instruction for atomic_subtract, but the subtraction will > still be atomic.] Essentially, the (old) value is available for > free, so why not provide it? It might even make sense to always > provide the old value for add/subtract, and have gcc throw away > the unused output (unless the input value is reused, when we'd > lose a whole register to the xadd). This might very well be a good idea to do. Having each of the atomic ops that currently return void return the new value (probably easiest to do that). The ia64 has a 'fetchadd' instruction for example. > Even if the processor does not support xadd-like operations, it > can be emulated (more expensively) using load, add, cmpxchg, loop- > if-failed [similar to the code frag below, but with a while() loop.] > But by providing a primitive, it can be optimized much further > than just C code using an atomic cmpxchg operation (and is > "negative cost" on x86 -- faster than the non-atomic version). Yes, an atomic_cmpset() loop is how the atomic ops are performed on the ia64. > In Julian's acquire_writer, we need to do an atomic compare-and-swap > operation, instead of assuming two operations are atomic (because the > above acquire_reader code could be executed between the two following > statements): > > => if ((ngq->q_flags & (~SINGLE_THREAD_ONLY)) == 0) { > => atomic_add_long(&ngq->q_flags, WRITER_ACTIVE); > > Here is a possible code sequence to "just get it working" (at least > I *think* this fixes the alleged problem): > [ register int flags; ] > => flags = ngq->q_flags; > => if ((flags & (~SINGLE_THREAD_ONLY) == 0) && > => atomic_cmpset(&ngq->q_flags, flags, flags + WRITER_ACTIVE)) { Yes, this does look correct. > One more thought on atomic operations: If we don't assume assignments > are atomic, and always use atomic_load and atomic_store, then we a) can > easily provide atomic 64-bit operations on x86 (quick hack would be > to use a single mutex for all 64-bit operations), and b) we can port > to platforms where atomic_add requires a mutex to protect the atomic_add > or atomic_cmpset sequence. [Slow as molasses] On x86, the load/store > macros are NOPs, but the use also (c) makes it clear that we are > manipulating a variable we perform atomic operations on. Note that the only atomic_load and atomic_store primities are those that include memory barriers (and I think they are broken on the x86 for that matter; they need to use a lock'd cmpxchgl in the load case and a lock'd xchgl in the store case I think.) > Kevin -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 13:43:22 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 13:43:18 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fast.cs.utah.edu (fast.cs.utah.edu [155.99.212.1]) by hub.freebsd.org (Postfix) with ESMTP id B6C1237B400; Tue, 2 Jan 2001 13:43:17 -0800 (PST) Received: (from vanmaren@localhost) by fast.cs.utah.edu (8.9.1/8.9.1) id OAA08265; Tue, 2 Jan 2001 14:43:10 -0700 (MST) Date: Tue, 2 Jan 2001 14:43:10 -0700 (MST) From: Kevin Van Maren Message-Id: <200101022143.OAA08265@fast.cs.utah.edu> To: jhb@FreeBSD.ORG, vanmaren@fast.cs.utah.edu Subject: Re: atomic increment? Cc: cp@bsdi.com, smp@FreeBSD.ORG Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > There is also a desire to try and keep the atomic API from being too huge I > think. Atomic operations are _expensive_. One thing you are forgetting about > on the x86 is that an atomic op on an SMP system requires a 'lock' prefix. The > cost of locking the bus drowns out the savings you may get by getting one or > two less instructions. atomic_increment/decrement doesn't inflate it *that* much. For most arch's it can point to atomic_add. I prefer ++ to +=1 for things that go up/down by one. Atomic_add of 1 may make more sense for some behaviors, but not for others. Believe me, I'm not forgetting the LOCK prefix. But LOCKing the memory bus is only the case on processors before the Pentium Pro. On the P6+, the LOCK signal is normally NOT asserted on the bus (for cacheable memory locations). However, it is still "expensive" because it is a memory fence, which can stall the speculative processor core. But it should be much less expensive than a cache miss in any case. [Yes, even on the P6, the extra cost will be larger that a single simple instruction in the cache. But just because we spend 30% of our time doing something doesn't mean we can't make the code faster by eliminating 20% elsewhere. But I'm NOT advocating the atomic_increment as a way to get around the gcc bug that causes it to move an immediate into a temp register: that performance boost should come from fixing gcc.] For many uses, atomic ops are better than a mutex, because: -) I have to acquire the memory I'm modifying an an exclusive state, even without the use of atomic operations. On a P6, the incremental cost for using an atomic operation is basically just incurred in preventing speculative reads to pass the op. -) If I use a mutex, I have to perform at least two LOCKed operations (one to acquire, one to release). Plus, not only do I have to obtain exclusive ownership of the variable, I also have to acquire exclusive ownership of the mutex cacheline. [If the mutex is in the same cacheline as the variable, and it isn't highly contested, the incremental cost is very low.] -) Performance comparisons are still assuming that the mutex isn't already owned, because if it is, the expense goes through the roof, either in spinning on the mutex, or in blocking. [Yes, other platforms may have different relative performance issues.] If you want to see slow, statically predict the wrong branch on IA64... I think it was something like an extra 50 cycles per loop iteration. > > One more thought on atomic operations: If we don't assume assignments > > are atomic, and always use atomic_load and atomic_store, then we a) can > > easily provide atomic 64-bit operations on x86 (quick hack would be > > to use a single mutex for all 64-bit operations), and b) we can port > > to platforms where atomic_add requires a mutex to protect the atomic_add > > or atomic_cmpset sequence. [Slow as molasses] On x86, the load/store > > macros are NOPs, but the use also (c) makes it clear that we are > > manipulating a variable we perform atomic operations on. > > Note that the only atomic_load and atomic_store primities are those that > include memory barriers (and I think they are broken on the x86 for that > matter; they need to use a lock'd cmpxchgl in the load case and a lock'd xchgl > in the store case I think.) The macros do (aligned) atomic assignments properly, but without the acquire and release semantics provided by LOCK: it is currently incorrect to release a lock by doing an "atomic_store(&lock, 0)". For that to be allowed, they would have to use x LOCKed xchg or something, as you say. From "man 9 atomic": The first form just performs the operation without any explicit barriers. The second form uses a read memory barrier, and the final variant uses a write memory barrier. The atomic_load() functions always have acquire semantics. Which is not the case for the current IA32 code (reads can pass unLOCKed reads). The atomic_store() functions always have release semantics. Which is not the case for the current IA32 code (reads can pass unLOCKed writes). I think separate memory barrier operations might also make sense. On IA64, we have the "mf" and "mf.a" operation; on IA32, we can emulate it by "lock; addl $0,0(%esp)" for SMP, and a NOP for uniprocessors (which is as free as you can make it, since (%esp) had better be exclusivly owned by the CPU). Memory barrier ops could be used under the "enhanced" load/store primitives, but they would be slower than direct encoded versions. I wonder how many problems we're going to have going from x86, where all LOCKed atomic operations have acquire and release semantics, to IA64, where they must be explicitly encoded. I guess it depends on how many places we use atomic primitives. Kevin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 17:10: 9 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 17:10:06 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from meow.osd.bsdi.com (meow.osd.bsdi.com [204.216.28.88]) by hub.freebsd.org (Postfix) with ESMTP id E3E6F37B402 for ; Tue, 2 Jan 2001 17:10:05 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by meow.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id f0319DG08578; Tue, 2 Jan 2001 17:09:13 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200101022143.OAA08265@fast.cs.utah.edu> Date: Tue, 02 Jan 2001 17:09:39 -0800 (PST) From: John Baldwin To: Kevin Van Maren Subject: Re: atomic increment? Cc: smp@FreeBSD.org, cp@bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 02-Jan-01 Kevin Van Maren wrote: >> > One more thought on atomic operations: If we don't assume assignments >> > are atomic, and always use atomic_load and atomic_store, then we a) can >> > easily provide atomic 64-bit operations on x86 (quick hack would be >> > to use a single mutex for all 64-bit operations), and b) we can port >> > to platforms where atomic_add requires a mutex to protect the atomic_add >> > or atomic_cmpset sequence. [Slow as molasses] On x86, the load/store >> > macros are NOPs, but the use also (c) makes it clear that we are >> > manipulating a variable we perform atomic operations on. >> >> Note that the only atomic_load and atomic_store primities are those that >> include memory barriers (and I think they are broken on the x86 for that >> matter; they need to use a lock'd cmpxchgl in the load case and a lock'd >> xchgl >> in the store case I think.) > > The macros do (aligned) atomic assignments properly, but without the > acquire and release semantics provided by LOCK: it is currently > incorrect to release a lock by doing an "atomic_store(&lock, 0)". > For that to be allowed, they would have to use x LOCKed xchg or > something, as you say. You can't do an atomic_store(&foo), you have to use atomic_store_rel(&foo), which is what I tried to say in the manpage. I guess that didn't come across clearly. > From "man 9 atomic": > The first form just performs the operation without any explicit barriers. > The second form uses a read memory barrier, and the final variant uses a > write memory barrier. The store and load don't have all of the forms. The load just has the acquire form, and the store just has the release form. > I think separate memory barrier operations might also make sense. > On IA64, we have the "mf" and "mf.a" operation; on IA32, we > can emulate it by "lock; addl $0,0(%esp)" for SMP, and a NOP for > uniprocessors (which is as free as you can make it, since (%esp) > had better be exclusivly owned by the CPU). Memory barrier ops > could be used under the "enhanced" load/store primitives, but > they would be slower than direct encoded versions. > > I wonder how many problems we're going to have going from x86, where > all LOCKed atomic operations have acquire and release semantics, to > IA64, where they must be explicitly encoded. I guess it depends on > how many places we use atomic primitives. Things that use atomic operations and depend on ordering should already be using the 'acq' and 'rel' variants of the atomic operations. See the mutex micro-operations for example that use atomic_cmpset_acq_ptr() and atomic_store_rel_ptr(). > Kevin -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 20: 5:36 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 20:05:33 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fast.cs.utah.edu (fast.cs.utah.edu [155.99.212.1]) by hub.freebsd.org (Postfix) with ESMTP id 4AEFB37B402; Tue, 2 Jan 2001 20:05:33 -0800 (PST) Received: (from vanmaren@localhost) by fast.cs.utah.edu (8.9.1/8.9.1) id VAA16941; Tue, 2 Jan 2001 21:05:32 -0700 (MST) Date: Tue, 2 Jan 2001 21:05:32 -0700 (MST) From: Kevin Van Maren Message-Id: <200101030405.VAA16941@fast.cs.utah.edu> To: jhb@freebsd.org Subject: Re: atomic increment? Cc: smp@freebsd.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org John, > > Note that the only atomic_load and atomic_store primities are those that > > include memory barriers (and I think they are broken on the x86 for that > > matter; they need to use a lock'd cmpxchgl in the load case and a lock'd xchgl > > in the store case I think.) Okay, I finally got it! These are also in the man page: > The atomic_load() functions always have acquire semantics. along with an example using atomic_load() instead of atomic_load_acq(). > The atomic_store() functions always have release semantics. So while atomic_load() is used in an example in the man page, there is no prototype for atomic_load/atomic_store, nor is there an implementation. You are saying that the ONLY form is atomic_store_rel instead of atomic_store (which doesn't exist). I think the man page could use some minor clarification. My point about the acquire/release "bugs" is that incorrect code will work on IA32 because of the stricter memory ordering guarantees. Getting every use correct is non-trivial, especially if there are a lot of them, or if other code changes around the atomic op. It is certainly easiest to program if all atomic ops have acquire/release semantics, but non-optimally-performing on IA64. So I guess that is one more thing against the use of atomic operations. load/cmpxchg code shouldn't be too hard: you just need a scratch register, set it equal to eax (and ANY garbage value), and LOCK cmpxchg it with the address. The read value is placed in (part of, for 8/16 bit ops) eax. One register-register mov and one atomic memory RMW cycle (which works because Intel always does the RMW cycle; it writes back the original value if the cmp fails, and eax will always contain the value that was in memory). Should be a pretty efficient inline asm. Do you want me to send in a patch? At least the ia64 atomic code does "the right thing" with acquire/release semantics on load/store. Kevin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 20:49:10 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 20:49:07 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pozo.com (pozo.com [216.101.162.50]) by hub.freebsd.org (Postfix) with ESMTP id 28C9637B400; Tue, 2 Jan 2001 20:49:07 -0800 (PST) Received: from dual.pozo.com (dual.pozo.com [216.101.162.51]) by pozo.com (8.11.1/8.11.1) with ESMTP id f034n4b00287; Tue, 2 Jan 2001 20:49:05 -0800 (PST) (envelope-from null@pozo.com) Message-Id: <5.0.2.1.2.20010102204401.00b13e88@pozo.com> X-Sender: null@pozo.com X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Tue, 02 Jan 2001 20:49:04 -0800 To: smp@freebsd.org From: Manfred Antar Subject: Fatal trap while printing under SMP Cc: current@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org When trying to print using a current SMP kernel, I get the following: Fatal trap 12: page fault while in kernel mode cpuid = 1; lapic.id = 0c000000 fault virtual address = 0xe1810412 fault code = supervisor write, page not present instruction pointer = 0x8:0xcb0a7977 stack pointer = 0x10:0xcb08cf84 frame pointer = 0x10:0xcb08cf9c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 30652 (irq7: lpt0) trap number = 12 panic: page fault cpuid = 1; lapic.id = 0c000000 boot() called on cpu#1 syncing disks... Printing works fine with a current non SMP kernel. Manfred ================================== || null@pozo.com || || Ph. (415) 681-6235 || ================================== To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 21: 5: 9 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 21:05:05 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id D1AC137B400; Tue, 2 Jan 2001 21:05:04 -0800 (PST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id D68656A90D; Wed, 3 Jan 2001 15:34:59 +1030 (CST) Date: Wed, 3 Jan 2001 15:34:59 +1030 From: Greg Lehey To: Manfred Antar Cc: smp@freebsd.org, current@freebsd.org Subject: Re: Fatal trap while printing under SMP Message-ID: <20010103153459.J4336@wantadilla.lemis.com> References: <5.0.2.1.2.20010102204401.00b13e88@pozo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <5.0.2.1.2.20010102204401.00b13e88@pozo.com>; from null@pozo.com on Tue, Jan 02, 2001 at 08:49:04PM -0800 Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tuesday, 2 January 2001 at 20:49:04 -0800, Manfred Antar wrote: > When trying to print using a current SMP kernel, I get the following: > > Fatal trap 12: page fault while in kernel mode > cpuid = 1; lapic.id = 0c000000 > fault virtual address = 0xe1810412 > fault code = supervisor write, page not present > instruction pointer = 0x8:0xcb0a7977 > stack pointer = 0x10:0xcb08cf84 > frame pointer = 0x10:0xcb08cf9c > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 30652 (irq7: lpt0) > trap number = 12 > panic: page fault > cpuid = 1; lapic.id = 0c000000 > boot() called on cpu#1 We really need more information than this. Can you get a dump, or at least a stack trace? Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 21:31:41 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 21:31:39 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from lolita.speakeasy.net (lolita.speakeasy.net [216.254.0.13]) by hub.freebsd.org (Postfix) with SMTP id 1409137B400 for ; Tue, 2 Jan 2001 21:31:39 -0800 (PST) Received: (qmail 7612 invoked from network); 3 Jan 2001 05:24:51 -0000 Received: from unknown (HELO gonzo.speakeasy.net) (192.168.0.5) by 192.168.0.13 with SMTP; 3 Jan 2001 05:24:51 -0000 Received: (qmail 21575 invoked from network); 3 Jan 2001 05:31:37 -0000 Received: from unknown (HELO celebris.tddhome) (64.81.20.229) by gonzo.speakeasy.net with SMTP; 3 Jan 2001 05:31:37 -0000 Received: (from tomdean@localhost) by celebris.tddhome (8.11.1/8.11.1) id f035Va418606; Tue, 2 Jan 2001 21:31:36 -0800 (PST) (envelope-from tomdean@speakeasy.org) Date: Tue, 2 Jan 2001 21:31:36 -0800 (PST) Message-Id: <200101030531.f035Va418606@celebris.tddhome> X-Authentication-Warning: celebris.tddhome: tomdean set sender to tomdean@speakeasy.org using -f From: "Thomas D. Dean" To: smp@FreeBSD.ORG In-reply-to: <5.0.2.1.2.20010102204401.00b13e88@pozo.com> (message from Manfred Antar on Tue, 02 Jan 2001 20:49:04 -0800) Subject: Re: Fatal trap while printing under SMP References: <5.0.2.1.2.20010102204401.00b13e88@pozo.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org This is the problem I reported a week, or so, ago. I still see it. The problem is to get more information so the developers can fix it. I don't know how to do that. tomdean To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 21:37:17 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 21:37:13 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from lolita.speakeasy.net (lolita.speakeasy.net [216.254.0.13]) by hub.freebsd.org (Postfix) with SMTP id BDA7437B400 for ; Tue, 2 Jan 2001 21:37:12 -0800 (PST) Received: (qmail 20181 invoked from network); 3 Jan 2001 05:30:25 -0000 Received: from unknown (HELO gonzo.speakeasy.net) (192.168.0.5) by 192.168.0.13 with SMTP; 3 Jan 2001 05:30:25 -0000 Received: (qmail 31438 invoked from network); 3 Jan 2001 05:37:08 -0000 Received: from unknown (HELO celebris.tddhome) (64.81.20.229) by gonzo.speakeasy.net with SMTP; 3 Jan 2001 05:37:08 -0000 Received: (from tomdean@localhost) by celebris.tddhome (8.11.1/8.11.1) id f035b6u18627; Tue, 2 Jan 2001 21:37:06 -0800 (PST) (envelope-from tomdean@speakeasy.org) Date: Tue, 2 Jan 2001 21:37:06 -0800 (PST) Message-Id: <200101030537.f035b6u18627@celebris.tddhome> X-Authentication-Warning: celebris.tddhome: tomdean set sender to tomdean@speakeasy.org using -f From: "Thomas D. Dean" To: grog@lemis.com Cc: null@pozo.com, smp@FreeBSD.ORG, current@FreeBSD.ORG In-reply-to: <20010103153459.J4336@wantadilla.lemis.com> (message from Greg Lehey on Wed, 3 Jan 2001 15:34:59 +1030) Subject: Re: Fatal trap while printing under SMP References: <5.0.2.1.2.20010102204401.00b13e88@pozo.com> <20010103153459.J4336@wantadilla.lemis.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Here is what I saw on 12/24. This is with DDB and KTRACE. I added options WITNESS options INVARIANT_SUPPORT options INVARIANTS but gained no futher information. Fatal Trap 12: page fault while in kernel mode cpuid=0; lapic.id=00000000 fault virtual address = 0x18c7a2bb fault code = 0x8:0xc6bb0ca0 stack pointer = 0x10:0xc7aa1f84 frame pointer = 0x10:0xc7aa1f9c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor flags = interrupt enabled, resume, IOPL=0 current process = 416 (irq7, lpt0) kernel: type 12 trap, code=0 cpu0 stopping CPUs: 0x00000002... stopped stopped at 0xc6bb0ca0: movb 0x18c7a22b, %al db> trace _end(0) at 0xc6bb0ca0 fork_trampoline() at fork_trampoline+0x45 tomdean To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Jan 2 21:41:38 2001 From owner-freebsd-smp@FreeBSD.ORG Tue Jan 2 21:41:36 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 9CEC837B400 for ; Tue, 2 Jan 2001 21:41:34 -0800 (PST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id 6D6D66A90D; Wed, 3 Jan 2001 16:11:31 +1030 (CST) Date: Wed, 3 Jan 2001 16:11:31 +1030 From: Greg Lehey To: Manfred Antar Cc: FreeBSD SMP list Subject: Re: Fatal trap while printing under SMP Message-ID: <20010103161131.K4336@wantadilla.lemis.com> References: <5.0.2.1.2.20010102204401.00b13e88@pozo.com> <5.0.2.1.2.20010102204401.00b13e88@pozo.com> <20010103153459.J4336@wantadilla.lemis.com> <5.0.2.1.2.20010102210933.046d3d70@pozo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <5.0.2.1.2.20010102210933.046d3d70@pozo.com>; from null@pozo.com on Tue, Jan 02, 2001 at 09:17:24PM -0800 Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tuesday, 2 January 2001 at 21:17:24 -0800, Manfred Antar wrote: > At 03:34 PM 1/3/2001 +1030, you wrote: >> On Tuesday, 2 January 2001 at 20:49:04 -0800, Manfred Antar wrote: >>> When trying to print using a current SMP kernel, I get the following: >>> >>> Fatal trap 12: page fault while in kernel mode >>> cpuid = 1; lapic.id = 0c000000 >>> fault virtual address = 0xe1810412 >>> fault code = supervisor write, page not present >>> instruction pointer = 0x8:0xcb0a7977 >>> stack pointer = 0x10:0xcb08cf84 >>> frame pointer = 0x10:0xcb08cf9c >>> code segment = base 0x0, limit 0xfffff, type 0x1b >>> = DPL 0, pres 1, def32 1, gran 1 >>> processor eflags = interrupt enabled, resume, IOPL = 0 >>> current process = 30652 (irq7: lpt0) >>> trap number = 12 >>> panic: page fault >>> cpuid = 1; lapic.id = 0c000000 >>> boot() called on cpu#1 >> >> We really need more information than this. Can you get a dump, or at >> least a stack trace? > > The machine freezes solid. You mean you can't even press Enter to get it to reboot? Bad news. > I have to do a reset to get it going again. I'm running it headless > off a serial port. I could hook a monitor and a keyboard to it and > try to get to the debugger but I have a feeling its going to be > wedged. How do i get a stack trace from debugger ? If you can get into the debugger, the ddb command is 't'. If you're doing remote serial debug with gdb, it's 'bt'. Unfortunately, it's not likely to help you much here. The IP value (cb0a7977) is almost certainly wrong. In other words, the real problem has already been lost. Under more normal circumstances, you could try this: # nm /boot/kernel/kernel | sort | less Then look for the fault IP address. Don't forget to remove the 0x at the beginning. But I strongly doubt you'll find anything. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jan 3 8:44:37 2001 From owner-freebsd-smp@FreeBSD.ORG Wed Jan 3 08:44:35 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id DD5C637B400 for ; Wed, 3 Jan 2001 08:44:34 -0800 (PST) Received: from foo.osd.bsdi.com (root@foo.osd.bsdi.com [204.216.28.137]) by pike.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id f03GiRh62514; Wed, 3 Jan 2001 08:44:27 -0800 (PST) (envelope-from jhb@foo.osd.bsdi.com) Received: (from jhb@localhost) by foo.osd.bsdi.com (8.11.1/8.11.0) id f03GiNQ49817; Wed, 3 Jan 2001 08:44:23 -0800 (PST) (envelope-from jhb) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200101030405.VAA16941@fast.cs.utah.edu> Date: Wed, 03 Jan 2001 08:44:23 -0800 (PST) Organization: BSD, Inc. From: John Baldwin To: Kevin Van Maren Subject: Re: atomic increment? Cc: smp@freebsd.org Sender: jhb@foo.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 03-Jan-01 Kevin Van Maren wrote: > John, > >> > Note that the only atomic_load and atomic_store primities are those that >> > include memory barriers (and I think they are broken on the x86 for that >> > matter; they need to use a lock'd cmpxchgl in the load case and a lock'd >> > xchgl >> > in the store case I think.) > > Okay, I finally got it! > > These are also in the man page: >> The atomic_load() functions always have acquire semantics. > along with an example using atomic_load() instead of atomic_load_acq(). > >> The atomic_store() functions always have release semantics. > > So while atomic_load() is used in an example in the man page, > there is no prototype for atomic_load/atomic_store, nor is there > an implementation. You are saying that the ONLY form is atomic_store_rel > instead of atomic_store (which doesn't exist). I think the man page > could use some minor clarification. Yes, it probably could. :) I'll see what I can come up with for that. > My point about the acquire/release "bugs" is that incorrect code > will work on IA32 because of the stricter memory ordering guarantees. > Getting every use correct is non-trivial, especially if there are a > lot of them, or if other code changes around the atomic op. It is > certainly easiest to program if all atomic ops have acquire/release > semantics, but non-optimally-performing on IA64. So I guess that > is one more thing against the use of atomic operations. One way to keep them more sane is to not use atomic ops all that much. Right now they are used in very few places. > load/cmpxchg code shouldn't be too hard: you just need a scratch > register, set it equal to eax (and ANY garbage value), and LOCK > cmpxchg it with the address. The read value is placed in (part of, > for 8/16 bit ops) eax. One register-register mov and one atomic > memory RMW cycle (which works because Intel always does the RMW > cycle; it writes back the original value if the cmp fails, and > eax will always contain the value that was in memory). Should > be a pretty efficient inline asm. > > Do you want me to send in a patch? If you'd like, sure. I was planning to use a lock'd xchgl for the store, but wasn't sure if that would work properly. Trying to use a cmpxchgl for the store would get ugly as you would have to do a loop until it actually did a store. Yuck. > At least the ia64 atomic code does "the right thing" with acquire/release > semantics on load/store. > > Kevin -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.Baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jan 3 10:39:34 2001 From owner-freebsd-smp@FreeBSD.ORG Wed Jan 3 10:39:30 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fast.cs.utah.edu (fast.cs.utah.edu [155.99.212.1]) by hub.freebsd.org (Postfix) with ESMTP id F3AC137B400; Wed, 3 Jan 2001 10:39:29 -0800 (PST) Received: (from vanmaren@localhost) by fast.cs.utah.edu (8.9.1/8.9.1) id LAA08694; Wed, 3 Jan 2001 11:39:29 -0700 (MST) Date: Wed, 3 Jan 2001 11:39:29 -0700 (MST) From: Kevin Van Maren Message-Id: <200101031839.LAA08694@fast.cs.utah.edu> To: jhb@freebsd.org, vanmaren@fast.cs.utah.edu Subject: Re: atomic increment? Cc: smp@freebsd.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > load/cmpxchg code shouldn't be too hard: you just need a scratch > > register, set it equal to eax (and ANY garbage value), and LOCK > > cmpxchg it with the address. The read value is placed in (part of, > > for 8/16 bit ops) eax. One register-register mov and one atomic > > memory RMW cycle (which works because Intel always does the RMW > > cycle; it writes back the original value if the cmp fails, and > > eax will always contain the value that was in memory). Should > > be a pretty efficient inline asm. > > > > Do you want me to send in a patch? > > If you'd like, sure. I was planning to use a lock'd xchgl for the store, but > wasn't sure if that would work properly. Trying to use a cmpxchgl for the > store would get ugly as you would have to do a loop until it actually did a > store. Yuck. For the store you'd want to do a straight xchgl, which will even return the old value (see my previous message about adding variants that return the old values...). We can even save a byte in the code by not using the LOCK prefix, since lock is always asserted for xchg. Hmmm. Actually, with writes being visible in-order on IA32, even on SMP, the store's release semantics may be okay as is? Okay, now I'M confusing myself. It won't provide memory fence semantics or strong ordering, but that isn't required by a store_rel. If the PPro retires instructions in order, and doesn't "execute" writes until they are retired, then the write can't complete before a previous read, so the current store code already has release semantics. Right? On IA64, a release doesn't have to occur before a following acquire; so we can allow a subsequent read to "pass" this store, as long as the CPU executes the store after all previos instructions. Certainly the read needs to be changed to provide acquire semantics. I'll look at writing a patch when I get off work tonight... Maybe I'll look at writing 64-bit operations using cmpxchg8b too (but that requires a Pentium -- and there were some dual 486 PCs built, not that we'd ever support the second CPU on them anyway). On uni-procs, we can disable interrupts and be okay. Kevin To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jan 3 10:50:53 2001 From owner-freebsd-smp@FreeBSD.ORG Wed Jan 3 10:50:50 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from meow.osd.bsdi.com (meow.osd.bsdi.com [204.216.28.88]) by hub.freebsd.org (Postfix) with ESMTP id D49DC37B400 for ; Wed, 3 Jan 2001 10:50:50 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by meow.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id f03InlG28097; Wed, 3 Jan 2001 10:49:47 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200101031839.LAA08694@fast.cs.utah.edu> Date: Wed, 03 Jan 2001 10:50:28 -0800 (PST) From: John Baldwin To: Kevin Van Maren Subject: Re: atomic increment? Cc: smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 03-Jan-01 Kevin Van Maren wrote: >> > load/cmpxchg code shouldn't be too hard: you just need a scratch >> > register, set it equal to eax (and ANY garbage value), and LOCK >> > cmpxchg it with the address. The read value is placed in (part of, >> > for 8/16 bit ops) eax. One register-register mov and one atomic >> > memory RMW cycle (which works because Intel always does the RMW >> > cycle; it writes back the original value if the cmp fails, and >> > eax will always contain the value that was in memory). Should >> > be a pretty efficient inline asm. >> > >> > Do you want me to send in a patch? >> >> If you'd like, sure. I was planning to use a lock'd xchgl for the store, >> but >> wasn't sure if that would work properly. Trying to use a cmpxchgl for the >> store would get ugly as you would have to do a loop until it actually did a >> store. Yuck. > > For the store you'd want to do a straight xchgl, which will even > return the old value (see my previous message about adding variants > that return the old values...). > We can even save a byte in the code by not using the LOCK prefix, since > lock is always asserted for xchg. Sounds good. > Hmmm. Actually, with writes being visible in-order on IA32, even on > SMP, the store's release semantics may be okay as is? Okay, now I'M > confusing myself. It won't provide memory fence semantics or strong > ordering, but that isn't required by a store_rel. If the PPro retires > instructions in order, and doesn't "execute" writes until they are > retired, then the write can't complete before a previous read, so the > current store code already has release semantics. Right? > On IA64, a release doesn't have to occur before a following acquire; > so we can allow a subsequent read to "pass" this store, as long as > the CPU executes the store after all previos instructions. store_rel still needs to act as a memory barrier to guarantee that any pending reads/writes are completed before the store in question so that all "protected" accesses are performed while the lock is still held. > Certainly the read needs to be changed to provide acquire semantics. Yes. > I'll look at writing a patch when I get off work tonight... > > Maybe I'll look at writing 64-bit operations using cmpxchg8b too (but > that requires a Pentium -- and there were some dual 486 PCs built, not > that we'd ever support the second CPU on them anyway). On uni-procs, > we can disable interrupts and be okay. You can assume a Pentium for the 64-bit stuff and use cmpxchg8b, I don't foresee us supporting MP on [34]86. This is something that has been sort of on my todo list, but I'm not entirely sure how useful it will be. > Kevin -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jan 3 19: 8:28 2001 From owner-freebsd-smp@FreeBSD.ORG Wed Jan 3 19:08:26 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from lolita.speakeasy.net (lolita.speakeasy.net [216.254.0.13]) by hub.freebsd.org (Postfix) with SMTP id C0AF837B400 for ; Wed, 3 Jan 2001 19:08:25 -0800 (PST) Received: (qmail 19712 invoked from network); 4 Jan 2001 03:01:11 -0000 Received: from unknown (HELO gonzo.speakeasy.net) (192.168.0.5) by 192.168.0.13 with SMTP; 4 Jan 2001 03:01:11 -0000 Received: (qmail 11668 invoked from network); 4 Jan 2001 03:08:00 -0000 Received: from unknown (HELO celebris.tddhome) (64.81.20.229) by gonzo.speakeasy.net with SMTP; 4 Jan 2001 03:08:00 -0000 Received: (from tomdean@localhost) by celebris.tddhome (8.11.1/8.11.1) id f04380L00892; Wed, 3 Jan 2001 19:08:00 -0800 (PST) (envelope-from tomdean@speakeasy.org) Date: Wed, 3 Jan 2001 19:08:00 -0800 (PST) Message-Id: <200101040308.f04380L00892@celebris.tddhome> X-Authentication-Warning: celebris.tddhome: tomdean set sender to tomdean@speakeasy.org using -f From: "Thomas D. Dean" To: freebsd-smp@FreeBSD.ORG Subject: Continuing Fatal Trap 12 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I tried -current SMP as of early this AM, PST. I still have the fatal trap 12 problem. I can generate it with a max of three transactions to lpd. The current process is (irq7: lpt0). trace shows only _end() and fork_trampoline. I posted the output of DDB a week ago. I have: options DDB options INVARIANT_SUPPORT options INVARIANTS options WITNESS options KTRACE How can I produce a stack trace, etc., so I can help find this problem? tomdean To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Jan 3 22: 1:34 2001 From owner-freebsd-smp@FreeBSD.ORG Wed Jan 3 22:01:31 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from io.yi.org (cr66388-a.rchrd1.on.wave.home.com [24.114.165.24]) by hub.freebsd.org (Postfix) with ESMTP id ADF2337B400 for ; Wed, 3 Jan 2001 22:01:30 -0800 (PST) Received: from io.yi.org (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by io.yi.org (Postfix) with ESMTP id 1D055BA7D; Thu, 4 Jan 2001 00:56:45 -0500 (EST) X-Mailer: exmh version 2.1.1 10/15/1999 To: "Thomas D. Dean" Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Continuing Fatal Trap 12 In-Reply-To: Message from "Thomas D. Dean" of "Wed, 03 Jan 2001 19:08:00 PST." <200101040308.f04380L00892@celebris.tddhome> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 04 Jan 2001 00:56:44 -0500 From: Jake Burkholder Message-Id: <20010104055645.1D055BA7D@io.yi.org> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > I tried -current SMP as of early this AM, PST. > > I still have the fatal trap 12 problem. I can generate it with a max > of three transactions to lpd. > > The current process is (irq7: lpt0). trace shows only _end() and > fork_trampoline. I posted the output of DDB a week ago. > > I have: > options DDB > options INVARIANT_SUPPORT > options INVARIANTS > options WITNESS > options KTRACE > > How can I produce a stack trace, etc., so I can help find this problem? Hi, I don't own a printer so its difficult for me to help, but having browsed the driver source there are a couple things that might be useful. I think you said before you can print fine with a uni-processor kernel, is that correct? Please send me the output of dmesg. Sorry if you posted it before, I can't find it. If you have a serial console, please compile a kernel with LPT_DEBUG, save the console messages to a file and send them to me. Include everything from the initial boot messages. To turn on the debugging go into /sys/dev/ppbus/lpt.c and add the line #define LPT_DEBUG anywhere in the file. Compile a new kernel and print stuff till it crashes. It will probably generate huge amounts of output. A possible workaround is to try to force the driver to use polling mode, since its crashing in the interrupt thread. To do this go into lpt.c again and make the following changes: --- lpt.c Thu Dec 7 17:33:12 2000 +++ lpt.c.hack Thu Jan 4 00:46:41 2001 @@ -394,6 +394,7 @@ /* retrieve the ppbus irq */ BUS_READ_IVAR(ppbus, dev, PPBUS_IVAR_IRQ, &irq); +#if 0 if (irq > 0) { /* declare our interrupt handler */ sc->intr_resource = bus_alloc_resource(dev, SYS_RES_IRQ, @@ -403,9 +404,12 @@ sc->sc_irq = LP_HAS_IRQ | LP_USE_IRQ | LP_ENABLE_IRQ; device_printf(dev, "Interrupt-driven port\n"); } else { +#endif sc->sc_irq = 0; device_printf(dev, "Polled port\n"); +#if 0 } +#endif lprintf(("irq %x %x\n", irq, sc->sc_irq)); lpt_release_ppbus(dev); It should say "Polled port" in dmesg instead of "Interrupt-driven port". Compile a new kernel and print some more stuff. The console output of a print job in polled mode with LPT_DEBUG enabled would also be helpful. This should work, but as I said I can't test it. Thanks Jake To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jan 4 7:12:56 2001 From owner-freebsd-smp@FreeBSD.ORG Thu Jan 4 07:12:55 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.gmx.net (pop.gmx.net [194.221.183.20]) by hub.freebsd.org (Postfix) with SMTP id 8B6D037B402 for ; Thu, 4 Jan 2001 07:12:54 -0800 (PST) Received: (qmail 6338 invoked by uid 0); 4 Jan 2001 15:12:52 -0000 Received: from pec-139-237.tnt8.hh2.uunet.de (HELO newton.gmx.de) (149.225.139.237) by mail.gmx.net (mail01) with SMTP; 4 Jan 2001 15:12:52 -0000 Message-Id: <4.3.1.0.20010104160928.00a6dbc0@pop.gmx.de> X-Sender: 7350523@pop.gmx.de X-Mailer: QUALCOMM Windows Eudora Version 4.3.1 Date: Thu, 04 Jan 2001 16:11:30 +0100 To: smp@freebsd.org From: Stefan Goltz Subject: subscribe freebsd-smp Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org subscribe freebsd-smp To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jan 4 21:57: 7 2001 From owner-freebsd-smp@FreeBSD.ORG Thu Jan 4 21:57:02 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pozo.com (pozo.com [216.101.162.50]) by hub.freebsd.org (Postfix) with ESMTP id D2C0837B400; Thu, 4 Jan 2001 21:57:01 -0800 (PST) Received: from dual.pozo.com (dual.pozo.com [216.101.162.51]) by pozo.com (8.11.1/8.11.1) with ESMTP id f055tvl01980; Thu, 4 Jan 2001 21:55:57 -0800 (PST) (envelope-from null@pozo.com) Message-Id: <5.0.2.1.2.20010104215541.00a63c38@pozo.com> X-Sender: null@pozo.com X-Mailer: QUALCOMM Windows Eudora Version 5.0.2 Date: Thu, 04 Jan 2001 21:55:57 -0800 To: Jake Burkholder From: Manfred Antar Subject: Re: Continuing Fatal Trap 12 Cc: current@freebsd.org;, smp@freebsd.org;, tomdean@speakeasy.org Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Jake I applied the patch: #define LPT_DEBUG=20 =20 --- lpt.c Thu Dec 7 17:33:12 2000=20 +++ lpt.c.hack Thu Jan 4 00:46:41 2001=20 @@ -394,6 +394,7 @@=20 /* retrieve the ppbus irq */=20 BUS_READ_IVAR(ppbus, dev, PPBUS_IVAR_IRQ, &irq);=20 =20 +#if 0=20 if (irq > 0) {=20 /* declare our interrupt handler */=20 sc->intr_resource =3D bus_alloc_resource(dev, SYS_RES_IRQ,=20 @@ -403,9 +404,12 @@=20 sc->sc_irq =3D LP_HAS_IRQ | LP_USE_IRQ | LP_ENABLE_IRQ;=20 device_printf(dev, "Interrupt-driven port\n");=20 } else {=20 +#endif=20 sc->sc_irq =3D 0;=20 device_printf(dev, "Polled port\n");=20 +#if 0=20 }=20 +#endif=20 lprintf(("irq %x %x\n", irq, sc->sc_irq));=20 =20 lpt_release_ppbus(dev);=20 And now printing seems to work fine under SMP Just alot of debbuging stuff when I print : Jan 4 21:52:45 pozo /boot/kernel/kernel:= ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp= pppppppppppppppp Jan 4 21:52:45 pozo /boot/kernel/kernel: pppppppppclosed. Jan 4 21:52:45 pozo /boot/kernel/kernel: pppppppppclosed. Manfred =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D || null@pozo.com || || Ph. (415) 681-6235 || =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jan 5 8: 4:16 2001 From owner-freebsd-smp@FreeBSD.ORG Fri Jan 5 08:04:13 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from lolita.speakeasy.net (lolita.speakeasy.net [216.254.0.13]) by hub.freebsd.org (Postfix) with SMTP id 14A8A37B402 for ; Fri, 5 Jan 2001 08:04:13 -0800 (PST) Received: (qmail 17533 invoked from network); 5 Jan 2001 15:57:16 -0000 Received: from unknown (HELO gonzo.speakeasy.net) (192.168.0.5) by 192.168.0.13 with SMTP; 5 Jan 2001 15:57:16 -0000 Received: (qmail 16845 invoked from network); 5 Jan 2001 16:04:11 -0000 Received: from unknown (HELO celebris.tddhome) (64.81.20.229) by gonzo.speakeasy.net with SMTP; 5 Jan 2001 16:04:11 -0000 Received: (from tomdean@localhost) by celebris.tddhome (8.11.1/8.11.1) id f05G4Af04311; Fri, 5 Jan 2001 08:04:10 -0800 (PST) (envelope-from tomdean@speakeasy.org) Date: Fri, 5 Jan 2001 08:04:10 -0800 (PST) Message-Id: <200101051604.f05G4Af04311@celebris.tddhome> X-Authentication-Warning: celebris.tddhome: tomdean set sender to tomdean@speakeasy.org using -f From: "Thomas D. Dean" To: smp@freebsd.org; In-reply-to: <5.0.2.1.2.20010104215541.00a63c38@pozo.com> (message from Manfred Antar on Thu, 04 Jan 2001 21:55:57 -0800) Subject: Re: Continuing Fatal Trap 12 References: <5.0.2.1.2.20010104215541.00a63c38@pozo.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Why does printing work correctly in UP but not SMP? I believe this patch masks a more serious problem in SMP. I still have system 'freezes' in SMP during 'make world'. In UP, 'make world' completes without error. Since printing seems to function normally in UP, the patch should be: --- lpt.c Thu Dec 7 17:33:12 2000 +++ lpt.c.hack Thu Jan 4 00:46:41 2001 @@ -394,6 +394,7 @@ /* retrieve the ppbus irq */ BUS_READ_IVAR(ppbus, dev, PPBUS_IVAR_IRQ, &irq); +#if !defined(SMP) if (irq > 0) { /* declare our interrupt handler */ sc->intr_resource = bus_alloc_resource(dev, SYS_RES_IRQ, @@ -403,9 +404,12 @@ sc->sc_irq = LP_HAS_IRQ | LP_USE_IRQ | LP_ENABLE_IRQ; device_printf(dev, "Interrupt-driven port\n"); } else { +#endif sc->sc_irq = 0; device_printf(dev, "Polled port\n"); +#if !defined(SMP) } +#endif lprintf(("irq %x %x\n", irq, sc->sc_irq)); lpt_release_ppbus(dev); tomdean To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jan 5 11: 6:58 2001 From owner-freebsd-smp@FreeBSD.ORG Fri Jan 5 11:06:54 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from meow.osd.bsdi.com (meow.osd.bsdi.com [204.216.28.88]) by hub.freebsd.org (Postfix) with ESMTP id AE2F037B699 for ; Fri, 5 Jan 2001 11:06:54 -0800 (PST) Received: from laptop.baldwin.cx (john@jhb-laptop.osd.bsdi.com [204.216.28.241]) by meow.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id f05J5fG86982; Fri, 5 Jan 2001 11:05:42 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200101051604.f05G4Af04311@celebris.tddhome> Date: Fri, 05 Jan 2001 11:06:50 -0800 (PST) From: John Baldwin To: "Thomas D. Dean" Subject: Re: Continuing Fatal Trap 12 Cc: smp@FreeBSD.org; Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 05-Jan-01 Thomas D. Dean wrote: > Why does printing work correctly in UP but not SMP? I believe this > patch masks a more serious problem in SMP. It looks like the stack is getting smashed for the interrupt thread in the SMP case. The patch Jake sent you is a test hack, not a real fix unfortunately. :( > I still have system 'freezes' in SMP during 'make world'. In UP, > 'make world' completes without error. So do I now on one of my SMP boxes. Debugging these is not easy unfortunately. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jan 5 11:44:36 2001 From owner-freebsd-smp@FreeBSD.ORG Fri Jan 5 11:44:34 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from lolita.speakeasy.net (lolita.speakeasy.net [216.254.0.13]) by hub.freebsd.org (Postfix) with SMTP id 4D2CA37B400 for ; Fri, 5 Jan 2001 11:44:33 -0800 (PST) Received: (qmail 4387 invoked from network); 5 Jan 2001 19:37:35 -0000 Received: from unknown (HELO gonzo.speakeasy.net) (192.168.0.5) by 192.168.0.13 with SMTP; 5 Jan 2001 19:37:35 -0000 Received: (qmail 28878 invoked from network); 5 Jan 2001 19:44:31 -0000 Received: from unknown (HELO celebris.tddhome) (64.81.20.229) by gonzo.speakeasy.net with SMTP; 5 Jan 2001 19:44:31 -0000 Received: (from tomdean@localhost) by celebris.tddhome (8.11.1/8.11.1) id f05JiTU04641; Fri, 5 Jan 2001 11:44:29 -0800 (PST) (envelope-from tomdean@speakeasy.org) Date: Fri, 5 Jan 2001 11:44:29 -0800 (PST) Message-Id: <200101051944.f05JiTU04641@celebris.tddhome> X-Authentication-Warning: celebris.tddhome: tomdean set sender to tomdean@speakeasy.org using -f From: "Thomas D. Dean" To: jhb@FreeBSD.org Cc: smp@FreeBSD.org; In-reply-to: (message from John Baldwin on Fri, 05 Jan 2001 11:06:50 -0800 (PST)) Subject: Re: Continuing Fatal Trap 12 References: Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi John, If you, or anyone can come up with a mechanism to test this, I can do it on this machine. Is there code I can insert into lpt.c to dump the stack? Would that possibly tell us who smashed it? This machine is not used for anything serious. I run UP to do email and minor net browsing. tomdean To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jan 5 13: 5: 6 2001 From owner-freebsd-smp@FreeBSD.ORG Fri Jan 5 13:05:02 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id E79E237B6B4 for ; Fri, 5 Jan 2001 13:05:01 -0800 (PST) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id f05L51o16204 for smp@freebsd.org; Fri, 5 Jan 2001 13:05:01 -0800 (PST) Date: Fri, 5 Jan 2001 13:05:01 -0800 From: Alfred Perlstein To: smp@freebsd.org Subject: [patch] cpuid on UP systems Message-ID: <20010105130500.M15744@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Sender: bright@fw.wintelcom.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I need cpuid to work on UP systems, this untested patch seems to compile ok, I'm wondering if I'm headed in the right direction here? John, Jake? Index: i386/machdep.c =================================================================== RCS file: /home/ncvs/src/sys/i386/i386/machdep.c,v retrieving revision 1.425 diff -u -r1.425 machdep.c --- i386/machdep.c 2000/12/13 10:00:42 1.425 +++ i386/machdep.c 2001/01/05 20:06:10 @@ -440,8 +440,9 @@ SLIST_INSERT_HEAD(&cpuhead, GLOBALDATA, gd_allcpu); mtx_init(&sched_lock, "sched lock", MTX_SPIN | MTX_COLD); - -#ifdef SMP +#ifndef SMP + GLOBALDATA->gd_cpuid = 0; +#else /* * OK, enough kmem_alloc/malloc state should be up, lets get on with it! */ Index: include/globaldata.h =================================================================== RCS file: /home/ncvs/src/sys/i386/include/globaldata.h,v retrieving revision 1.15 diff -u -r1.15 globaldata.h --- include/globaldata.h 2000/10/06 02:20:13 1.15 +++ include/globaldata.h 2001/01/05 20:00:02 @@ -65,8 +65,8 @@ struct segment_descriptor gd_common_tssd; struct segment_descriptor *gd_tss_gdt; int gd_currentldt; /* only used for USER_LDT */ -#ifdef SMP u_int gd_cpuid; +#ifdef SMP u_int gd_cpu_lockid; u_int gd_other_cpus; int gd_inside_intr; Index: include/globals.h =================================================================== RCS file: /home/ncvs/src/sys/i386/include/globals.h,v retrieving revision 1.13 diff -u -r1.13 globals.h --- include/globals.h 2000/11/15 22:02:05 1.13 +++ include/globals.h 2001/01/05 19:57:22 @@ -106,8 +106,8 @@ #define currentldt GLOBAL_RVALUE(currentldt, int) #endif -#ifdef SMP #define cpuid GLOBAL_RVALUE(cpuid, u_int) +#ifdef SMP #define other_cpus GLOBAL_LVALUE(other_cpus, u_int) #define inside_intr GLOBAL_LVALUE(inside_intr, int) #define prv_CMAP1 GLOBAL_LVALUE(prv_CMAP1, pt_entry_t *) -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jan 5 13:31:26 2001 From owner-freebsd-smp@FreeBSD.ORG Fri Jan 5 13:31:24 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from cr66388-a.rchrd1.on.wave.home.com (cr66388-a.rchrd1.on.wave.home.com [24.114.165.24]) by hub.freebsd.org (Postfix) with ESMTP id 19ED937B400 for ; Fri, 5 Jan 2001 13:31:24 -0800 (PST) Received: from cr66388-a.rchrd1.on.wave.home.c (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by cr66388-a.rchrd1.on.wave.home.com (Postfix) with ESMTP id E43C9BA7D; Fri, 5 Jan 2001 16:26:36 -0500 (EST) X-Mailer: exmh version 2.1.1 10/15/1999 To: Alfred Perlstein Cc: smp@FreeBSD.ORG, jake@cr66388-a.rchrd1.on.wave.home.com Subject: Re: [patch] cpuid on UP systems In-Reply-To: Message from Alfred Perlstein of "Fri, 05 Jan 2001 13:05:01 PST." <20010105130500.M15744@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Fri, 05 Jan 2001 16:26:36 -0500 From: Jake Burkholder Message-Id: <20010105212636.E43C9BA7D@cr66388-a.rchrd1.on.wave.home.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > I need cpuid to work on UP systems, this untested patch seems to > compile ok, I'm wondering if I'm headed in the right direction here? > > John, Jake? > No, please wait a little. I will do a major overhaul of the per-cpu variable stuff this weekend which will make cpuid and some other things ubiquitous. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jan 5 14:27:37 2001 From owner-freebsd-smp@FreeBSD.ORG Fri Jan 5 14:27:36 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 28FB637B400 for ; Fri, 5 Jan 2001 14:27:36 -0800 (PST) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id f05MRV618302; Fri, 5 Jan 2001 14:27:31 -0800 (PST) Date: Fri, 5 Jan 2001 14:27:31 -0800 From: Alfred Perlstein To: Jake Burkholder Cc: smp@FreeBSD.ORG, jake@cr66388-a.rchrd1.on.wave.home.com Subject: Re: [patch] cpuid on UP systems Message-ID: <20010105142731.R15744@fw.wintelcom.net> References: <20010105212636.E43C9BA7D@cr66388-a.rchrd1.on.wave.home.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20010105212636.E43C9BA7D@cr66388-a.rchrd1.on.wave.home.com>; from jake@cr66388-a.rchrd1.on.wave.home.c on Fri, Jan 05, 2001 at 04:26:36PM -0500 Sender: bright@fw.wintelcom.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org * Jake Burkholder [010105 13:31] wrote: > > I need cpuid to work on UP systems, this untested patch seems to > > compile ok, I'm wondering if I'm headed in the right direction here? > > > > John, Jake? > > > > No, please wait a little. I will do a major overhaul of the per-cpu > variable stuff this weekend which will make cpuid and some other things > ubiquitous. Er, ok, but I'm counting on it so I can begin testing/benching my new allocator. -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jan 5 14:36:55 2001 From owner-freebsd-smp@FreeBSD.ORG Fri Jan 5 14:36:53 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id A0A6537B400 for ; Fri, 5 Jan 2001 14:36:53 -0800 (PST) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id f05MarE18574; Fri, 5 Jan 2001 14:36:53 -0800 (PST) Date: Fri, 5 Jan 2001 14:36:53 -0800 From: Alfred Perlstein To: Jake Burkholder Cc: smp@FreeBSD.ORG, jake@cr66388-a.rchrd1.on.wave.home.com Subject: Re: [patch] cpuid on UP systems Message-ID: <20010105143653.S15744@fw.wintelcom.net> References: <20010105212636.E43C9BA7D@cr66388-a.rchrd1.on.wave.home.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20010105212636.E43C9BA7D@cr66388-a.rchrd1.on.wave.home.com>; from jake@cr66388-a.rchrd1.on.wave.home.c on Fri, Jan 05, 2001 at 04:26:36PM -0500 Sender: bright@fw.wintelcom.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org * Jake Burkholder [010105 13:31] wrote: > > I need cpuid to work on UP systems, this untested patch seems to > > compile ok, I'm wondering if I'm headed in the right direction here? > > > > John, Jake? > > > > No, please wait a little. I will do a major overhaul of the per-cpu > variable stuff this weekend which will make cpuid and some other things > ubiquitous. Also, what about hw.ncpu? There doesn't seem to be a nice way of getting this, basically if SMP is defined it's named something different than with it defined. -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Jan 6 12:17:59 2001 From owner-freebsd-smp@FreeBSD.ORG Sat Jan 6 12:17:58 2001 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from cr66388-a.rchrd1.on.wave.home.com (cr66388-a.rchrd1.on.wave.home.com [24.114.165.24]) by hub.freebsd.org (Postfix) with ESMTP id AE6B537B400 for ; Sat, 6 Jan 2001 12:17:57 -0800 (PST) Received: from cr66388-a.rchrd1.on.wave.home.c (localhost.gvcl1.bc.wave.home.com [127.0.0.1]) by cr66388-a.rchrd1.on.wave.home.com (Postfix) with ESMTP id 67AFABA7D; Sat, 6 Jan 2001 15:13:23 -0500 (EST) X-Mailer: exmh version 2.1.1 10/15/1999 To: Alfred Perlstein Cc: smp@FreeBSD.ORG Subject: Re: [patch] cpuid on UP systems In-Reply-To: Message from Alfred Perlstein of "Fri, 05 Jan 2001 14:27:31 PST." <20010105142731.R15744@fw.wintelcom.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sat, 06 Jan 2001 15:13:23 -0500 From: Jake Burkholder Message-Id: <20010106201323.67AFABA7D@cr66388-a.rchrd1.on.wave.home.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > * Jake Burkholder [010105 13:31] wrote: > > > I need cpuid to work on UP systems, this untested patch seems to > > > compile ok, I'm wondering if I'm headed in the right direction here? > > > > > > John, Jake? > > > > > > > No, please wait a little. I will do a major overhaul of the per-cpu > > variable stuff this weekend which will make cpuid and some other things > > ubiquitous. > > Er, ok, but I'm counting on it so I can begin testing/benching my > new allocator. > Ok, this is done. I think I forgot to initialize it for i386 UP though, so it might be garbage. Do something like __globaldata.gd_cpuid = 0 where its setup in machdep.c:init386(). I'll fix this as soon as I can. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message