From owner-freebsd-smp Sun Mar 31 9:29:11 2002 Delivered-To: freebsd-smp@freebsd.org Received: from wall.polstra.com (wall-gw.polstra.com [206.213.73.130]) by hub.freebsd.org (Postfix) with ESMTP id CEB0B37B419 for ; Sun, 31 Mar 2002 09:29:07 -0800 (PST) Received: from vashon.polstra.com (vashon.polstra.com [206.213.73.13]) by wall.polstra.com (8.11.3/8.11.3) with ESMTP id g2VHT5o95913; Sun, 31 Mar 2002 09:29:05 -0800 (PST) (envelope-from jdp@wall.polstra.com) Received: (from jdp@localhost) by vashon.polstra.com (8.11.6/8.11.0) id g2VHT5h18352; Sun, 31 Mar 2002 09:29:05 -0800 (PST) (envelope-from jdp) Date: Sun, 31 Mar 2002 09:29:05 -0800 (PST) Message-Id: <200203311729.g2VHT5h18352@vashon.polstra.com> To: smp@freebsd.org From: John Polstra Cc: dillon@apollo.backplane.com Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. In-Reply-To: <200203292207.g2TM7Fi67491@apollo.backplane.com> References: <200203292207.g2TM7Fi67491@apollo.backplane.com> Organization: Polstra & Co., Seattle, WA Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org In article <200203292207.g2TM7Fi67491@apollo.backplane.com>, Matthew Dillon wrote: > This won't save you from stalls. Whenever one cpu writes to > the cache, on IA32, it's write-through to main-memory which > invalidates all other cpu's caches for that cache line. No, it's not nearly that bad. See section 9.2 of the IA-32 Intel Architecture Software Developer's Manual, Volume 3 (24547203.pdf from developer.intel.com). It describes what happens in a couple of different scenarios, for example: Beginning with the P6 family processors, if a processor detects (through snooping) that another processor is trying to access a memory location that it has modified in its cache, but has not yet written back to system memory, the snooping processor will signal the other processor (by means of the HITM# signal) that the cache line is held in modified state and will preform an implicit write-back of the modified data. The implicit write-back is transferred directly to the initial requesting processor and snooped by the memory controller to assure that system memory has been updated. Here, the processor with the valid data may pass the data to the other proces-sors without actually writing it to system memory; however, it is the responsibility of the memory controller to snoop this operation and update memory. John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 9:47:57 2002 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id EB0FE37B41D for ; Sun, 31 Mar 2002 09:47:51 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.11.6/8.9.1) id g2VHlII89488; Sun, 31 Mar 2002 09:47:18 -0800 (PST) (envelope-from dillon) Date: Sun, 31 Mar 2002 09:47:18 -0800 (PST) From: Matthew Dillon Message-Id: <200203311747.g2VHlII89488@apollo.backplane.com> To: John Polstra Cc: smp@FreeBSD.ORG Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. References: <200203292207.g2TM7Fi67491@apollo.backplane.com> <200203311729.g2VHT5h18352@vashon.polstra.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org :> the cache, on IA32, it's write-through to main-memory which :> invalidates all other cpu's caches for that cache line. : :No, it's not nearly that bad. See section 9.2 of the IA-32 Intel :Architecture Software Developer's Manual, Volume 3 (24547203.pdf from :developer.intel.com). It describes what happens in a couple of :different scenarios, for example: This is not a case that typically occurs. Because intel caches are write-through the processor that does the write will flush the dirty cache line to main memory in fairly short order. I'm fairly sure that the flush causes all other processors holding that cache line to invalidate their cache line rather then snoop/update it. The case you describe below occurs when a processor does not have *ANY* data cached and attempts to read a memory location. Since it is leaving the cpu, this is a snoopable transaction. Any other cpu in the system that has the cache line in question can theoretically supply the data to the originating cpu rather then having to wait for the memory controller to supply it. The manual section you describe below indicates that Intel implements this at least when the data is dirty in some cpu's cache. You or I would have to read-up on the specs to determine whether it also occurs if the data is clean. In anycase, I ran real live tests. The stall is real and just as bad as if you had used a locked bus cycle. I don't know whether it is retrieving the data via snooping or via the memory controller but the stall is there either way. -Matt Matthew Dillon : Beginning with the P6 family processors, if a processor detects : (through snooping) that another processor is trying to access a : memory location that it has modified in its cache, but has not : yet written back to system memory, the snooping processor will : signal the other processor (by means of the HITM# signal) that : the cache line is held in modified state and will preform an : implicit write-back of the modified data. The implicit write-back : is transferred directly to the initial requesting processor and : snooped by the memory controller to assure that system memory has : been updated. Here, the processor with the valid data may pass : the data to the other proces-sors without actually writing it to : system memory; however, it is the responsibility of the memory : controller to snoop this operation and update memory. : :John :-- : John Polstra : John D. Polstra & Co., Inc. Seattle, Washington USA : "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 9:52:51 2002 Delivered-To: freebsd-smp@freebsd.org Received: from wall.polstra.com (wall-gw.polstra.com [206.213.73.130]) by hub.freebsd.org (Postfix) with ESMTP id D5AE137B405 for ; Sun, 31 Mar 2002 09:52:37 -0800 (PST) Received: from vashon.polstra.com (vashon.polstra.com [206.213.73.13]) by wall.polstra.com (8.11.3/8.11.3) with ESMTP id g2VHqao96095; Sun, 31 Mar 2002 09:52:36 -0800 (PST) (envelope-from jdp@wall.polstra.com) Received: (from jdp@localhost) by vashon.polstra.com (8.11.6/8.11.0) id g2VHqab18408; Sun, 31 Mar 2002 09:52:36 -0800 (PST) (envelope-from jdp) Date: Sun, 31 Mar 2002 09:52:36 -0800 (PST) Message-Id: <200203311752.g2VHqab18408@vashon.polstra.com> To: smp@freebsd.org From: John Polstra Cc: dillon@apollo.backplane.com Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. In-Reply-To: <200203311747.g2VHlII89488@apollo.backplane.com> References: <200203292207.g2TM7Fi67491@apollo.backplane.com> <200203311729.g2VHT5h18352@vashon.polstra.com> <200203311747.g2VHlII89488@apollo.backplane.com> Organization: Polstra & Co., Seattle, WA Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org In article <200203311747.g2VHlII89488@apollo.backplane.com>, Matthew Dillon wrote: > > :> the cache, on IA32, it's write-through to main-memory which > :> invalidates all other cpu's caches for that cache line. > : > :No, it's not nearly that bad. See section 9.2 of the IA-32 Intel > :Architecture Software Developer's Manual, Volume 3 (24547203.pdf from > :developer.intel.com). It describes what happens in a couple of > :different scenarios, for example: > > This is not a case that typically occurs. Because intel caches are > write-through the processor that does the write will flush the dirty > cache line to main memory in fairly short order. Why do you keep saying the Intel caches are write-through? They've been write-back since the Pentium. See table 9-2 in the same document I cited before. John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 10: 9:16 2002 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id EE3C237B41D for ; Sun, 31 Mar 2002 10:09:10 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.11.6/8.9.1) id g2VI90H89605; Sun, 31 Mar 2002 10:09:00 -0800 (PST) (envelope-from dillon) Date: Sun, 31 Mar 2002 10:09:00 -0800 (PST) From: Matthew Dillon Message-Id: <200203311809.g2VI90H89605@apollo.backplane.com> To: John Polstra Cc: smp@FreeBSD.ORG Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. References: <200203292207.g2TM7Fi67491@apollo.backplane.com> <200203311729.g2VHT5h18352@vashon.polstra.com> <200203311747.g2VHlII89488@apollo.backplane.com> <200203311752.g2VHqab18408@vashon.polstra.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org :> write-through the processor that does the write will flush the dirty :> cache line to main memory in fairly short order. : :Why do you keep saying the Intel caches are write-through? They've :been write-back since the Pentium. See table 9-2 in the same document :I cited before. : :John :-- : John Polstra Maybe I'm using the wrong terminology. What I mean to say is that Intel caches, under most conditions, will flush dirty elements in their caches to main memory very quickly. i.e. unlike, say, the old 68040 cache which leaves dirty cache lines in the cache almost indefinitely. The intel caches implement a write ordering constraint and a FIFO to deal with dirty data. Yes, I guess that would be write-back rather then write-through, But not delayed-write. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 10:17:20 2002 Delivered-To: freebsd-smp@freebsd.org Received: from wall.polstra.com (wall-gw.polstra.com [206.213.73.130]) by hub.freebsd.org (Postfix) with ESMTP id E73DA37B41C for ; Sun, 31 Mar 2002 10:17:15 -0800 (PST) Received: from vashon.polstra.com (vashon.polstra.com [206.213.73.13]) by wall.polstra.com (8.11.3/8.11.3) with ESMTP id g2VIHEo96427; Sun, 31 Mar 2002 10:17:14 -0800 (PST) (envelope-from jdp@wall.polstra.com) Received: (from jdp@localhost) by vashon.polstra.com (8.11.6/8.11.0) id g2VIHEB18544; Sun, 31 Mar 2002 10:17:14 -0800 (PST) (envelope-from jdp) Date: Sun, 31 Mar 2002 10:17:14 -0800 (PST) Message-Id: <200203311817.g2VIHEB18544@vashon.polstra.com> To: smp@freebsd.org From: John Polstra Cc: dillon@apollo.backplane.com Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. In-Reply-To: <200203311809.g2VI90H89605@apollo.backplane.com> References: <200203311747.g2VHlII89488@apollo.backplane.com> <200203311752.g2VHqab18408@vashon.polstra.com> <200203311809.g2VI90H89605@apollo.backplane.com> Organization: Polstra & Co., Seattle, WA Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org In article <200203311809.g2VI90H89605@apollo.backplane.com>, Matthew Dillon wrote: > : > :Why do you keep saying the Intel caches are write-through? They've > :been write-back since the Pentium. See table 9-2 in the same document > :I cited before. > : > Maybe I'm using the wrong terminology. What I mean to say is that > Intel caches, under most conditions, will flush dirty elements in > their caches to main memory very quickly. i.e. unlike, say, the old > 68040 cache which leaves dirty cache lines in the cache almost > indefinitely. The intel caches implement a write ordering constraint > and a FIFO to deal with dirty data. > > Yes, I guess that would be write-back rather then write-through, > But not delayed-write. It is both write-back and delayed-write. Section 9.10 ("Store Buffer") describes it: IA-32 processors temporarily store each write (store) to memory in a store buffer. The store buffer improves processor performance by allowing the processor to continue executing instructions without having to wait until a write to memory and/or to a cache is complete. It also allows writes to be delayed for more efficient use of memory-access bus cycles. Table 9-1 states that the store buffer has 24 entries on the Pentium 4, and 12 entries on the P6 family. John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 10:35: 3 2002 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 973A137B417 for ; Sun, 31 Mar 2002 10:34:55 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.11.6/8.9.1) id g2VIYrt89705; Sun, 31 Mar 2002 10:34:53 -0800 (PST) (envelope-from dillon) Date: Sun, 31 Mar 2002 10:34:53 -0800 (PST) From: Matthew Dillon Message-Id: <200203311834.g2VIYrt89705@apollo.backplane.com> To: John Polstra Cc: smp@FreeBSD.ORG Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. References: <200203311747.g2VHlII89488@apollo.backplane.com> <200203311752.g2VHqab18408@vashon.polstra.com> <200203311809.g2VI90H89605@apollo.backplane.com> <200203311817.g2VIHEB18544@vashon.polstra.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org :> But not delayed-write. : :It is both write-back and delayed-write. Section 9.10 ("Store :Buffer") describes it: : : IA-32 processors temporarily store each write (store) to memory in : a store buffer. The store buffer improves processor performance by : allowing the processor to continue executing instructions without : having to wait until a write to memory and/or to a cache is : complete. It also allows writes to be delayed for more efficient : use of memory-access bus cycles. : :Table 9-1 states that the store buffer has 24 entries on the Pentium 4, :and 12 entries on the P6 family. : :John :-- : John Polstra : John D. Polstra & Co., Inc. Seattle, Washington USA : "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa Now we are quibbling over terminology. Intel caches have a write FIFO. They are not a full-blown delayed-write caches. There is a BIG difference. That is, you can't have an arbitrary amount of dirty data sitting in an intel cache. This means that a write will be pushed out to main memory in fairly short order. The only way to prevent this from occuring is to saturate the memory bus with cache-missed reads. There is no confusion here, the linux folks did a massive amount of testing of this when they were trying to decide whether to force a flush after releasing a mutex or whether to allow the cpu to flush it out in the background. Tests showed that the cpu would normally flush the data out within a microsecond or two unless code was carefully crafted to saturate the memory bus with reads. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 10:41:43 2002 Delivered-To: freebsd-smp@freebsd.org Received: from wall.polstra.com (wall-gw.polstra.com [206.213.73.130]) by hub.freebsd.org (Postfix) with ESMTP id F09AF37B419 for ; Sun, 31 Mar 2002 10:41:39 -0800 (PST) Received: from vashon.polstra.com (vashon.polstra.com [206.213.73.13]) by wall.polstra.com (8.11.3/8.11.3) with ESMTP id g2VIfco96543; Sun, 31 Mar 2002 10:41:38 -0800 (PST) (envelope-from jdp@wall.polstra.com) Received: (from jdp@localhost) by vashon.polstra.com (8.11.6/8.11.0) id g2VIfcn18637; Sun, 31 Mar 2002 10:41:38 -0800 (PST) (envelope-from jdp) Date: Sun, 31 Mar 2002 10:41:38 -0800 (PST) Message-Id: <200203311841.g2VIfcn18637@vashon.polstra.com> To: smp@freebsd.org From: John Polstra Cc: dillon@apollo.backplane.com Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. In-Reply-To: <200203311834.g2VIYrt89705@apollo.backplane.com> References: <200203311809.g2VI90H89605@apollo.backplane.com> <200203311817.g2VIHEB18544@vashon.polstra.com> <200203311834.g2VIYrt89705@apollo.backplane.com> Organization: Polstra & Co., Seattle, WA Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org In article <200203311834.g2VIYrt89705@apollo.backplane.com>, Matthew Dillon wrote: > > Now we are quibbling over terminology. Intel caches have a write FIFO. > They are not a full-blown delayed-write caches. There is a BIG > difference. That is, you can't have an arbitrary amount of dirty data > sitting in an intel cache. There is nothing in the Intel documentation which would support that statement. The cache is write-back. Nowhere does the documentation say anything about dirty lines being flushed to memory except when required by the cache control protocol (which is pretty standard). > This means that a write will be pushed out to main memory in fairly > short order. No, I don't think so. There is no evidence of that in the Intel docs. John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 11: 2:26 2002 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 5865D37B417 for ; Sun, 31 Mar 2002 11:02:21 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.11.6/8.9.1) id g2VJ28i89787; Sun, 31 Mar 2002 11:02:08 -0800 (PST) (envelope-from dillon) Date: Sun, 31 Mar 2002 11:02:08 -0800 (PST) From: Matthew Dillon Message-Id: <200203311902.g2VJ28i89787@apollo.backplane.com> To: John Polstra Cc: smp@FreeBSD.ORG Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. References: <200203311809.g2VI90H89605@apollo.backplane.com> <200203311817.g2VIHEB18544@vashon.polstra.com> <200203311834.g2VIYrt89705@apollo.backplane.com> <200203311841.g2VIfcn18637@vashon.polstra.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org :> Now we are quibbling over terminology. Intel caches have a write FIFO. :> They are not a full-blown delayed-write caches. There is a BIG :> difference. That is, you can't have an arbitrary amount of dirty data :> sitting in an intel cache. : :There is nothing in the Intel documentation which would support that :statement. The cache is write-back. Nowhere does the documentation :say anything about dirty lines being flushed to memory except when :required by the cache control protocol (which is pretty standard). : :> This means that a write will be pushed out to main memory in fairly :> short order. : :No, I don't think so. There is no evidence of that in the Intel :docs. : :John :-- : John Polstra I can't argue against faith, I suppose, but you shouldn't be so quick to throw away the thousands of emails on the Linux and FreeBSD lists and at least a dozen people who have have tested this stuff on real live machines. The linux folks did some pretty severe testing, especially testing of non-flushed write-latencies in MP systems. I did some testing of this myself when I redid the low level MP locking code. All of it is available on the lists if you look. It doesn't magically go away because the documentation says so (or doesn't say so in this case). In otherwords, what I'm talking about here is the reality of the hardware, found through testing. When John brought up using a non-locked bus cycle, for example, I tested the case and found that it doesn't prevent stalls. If you want to assert that I am incorrect then I strongly recommend that you back those assertions up with your own tests showing otherwise. Also, I'm not sure why you seem to read so much into the Intel documentation. Intel documentation is notoriously incomplete in many respects and just plain wrong in others. For example, the linux folks have shown definitively that if MP systems do dirty-pending-write snooping at all, it doesn't help latencies in the least. I think that in Intel's entire documentation set they only talk about MP write ordering in one or two places. I don't remember where, but I do remember the linux folks having to dig fairly deeply to find it. They finally located an Intel engineer and asking him for the final word on the matter, in fact. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 11:14:36 2002 Delivered-To: freebsd-smp@freebsd.org Received: from wall.polstra.com (wall-gw.polstra.com [206.213.73.130]) by hub.freebsd.org (Postfix) with ESMTP id C24E837B400 for ; Sun, 31 Mar 2002 11:14:27 -0800 (PST) Received: from vashon.polstra.com (vashon.polstra.com [206.213.73.13]) by wall.polstra.com (8.11.3/8.11.3) with ESMTP id g2VJEPo96669; Sun, 31 Mar 2002 11:14:25 -0800 (PST) (envelope-from jdp@wall.polstra.com) Received: (from jdp@localhost) by vashon.polstra.com (8.11.6/8.11.0) id g2VJEPL18834; Sun, 31 Mar 2002 11:14:25 -0800 (PST) (envelope-from jdp) Date: Sun, 31 Mar 2002 11:14:25 -0800 (PST) Message-Id: <200203311914.g2VJEPL18834@vashon.polstra.com> To: smp@freebsd.org From: John Polstra Cc: dillon@apollo.backplane.com Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. In-Reply-To: <200203311902.g2VJ28i89787@apollo.backplane.com> References: <200203311834.g2VIYrt89705@apollo.backplane.com> <200203311841.g2VIfcn18637@vashon.polstra.com> <200203311902.g2VJ28i89787@apollo.backplane.com> Organization: Polstra & Co., Seattle, WA Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org In article <200203311902.g2VJ28i89787@apollo.backplane.com>, Matthew Dillon wrote: > > :> Now we are quibbling over terminology. Intel caches have a write FIFO. > :> They are not a full-blown delayed-write caches. There is a BIG > :> difference. That is, you can't have an arbitrary amount of dirty data > :> sitting in an intel cache. > : > :There is nothing in the Intel documentation which would support that > :statement. The cache is write-back. Nowhere does the documentation > :say anything about dirty lines being flushed to memory except when > :required by the cache control protocol (which is pretty standard). > : > :> This means that a write will be pushed out to main memory in fairly > :> short order. > : > :No, I don't think so. There is no evidence of that in the Intel > :docs. > : > I can't argue against faith, I suppose, You are not arguing against faith, you are arguing against the documentation. I provided references for the claims I made, but I haven't seen any references coming back the other way. > but you shouldn't be so quick > to throw away the thousands of emails on the Linux and FreeBSD lists > and at least a dozen people who have have tested this stuff on real > live machines. The linux folks did some pretty severe testing, > especially testing of non-flushed write-latencies in MP systems. I > did some testing of this myself when I redid the low level MP locking > code. All of it is available on the lists if you look. It doesn't > magically go away because the documentation says so (or doesn't say so > in this case). > > In otherwords, what I'm talking about here is the reality of the hardware, > found through testing. When John brought up using a non-locked bus cycle, > for example, I tested the case and found that it doesn't prevent stalls. > > If you want to assert that I am incorrect then I strongly recommend that > you back those assertions up with your own tests showing otherwise. I am not disputing your tests. I am disputing the conclusions you drew from them. > Also, I'm not sure why you seem to read so much into the Intel > documentation. Intel documentation is notoriously incomplete in many > respects and just plain wrong in others. For example, the linux folks > have shown definitively that if MP systems do dirty-pending-write snooping > at all, it doesn't help latencies in the least. I think that in Intel's > entire documentation set they only talk about MP write ordering in one > or two places. I don't remember where, but I do remember the linux > folks having to dig fairly deeply to find it. They finally located > an Intel engineer and asking him for the final word on the matter, > in fact. Bad example. I'm quite familiar with that thread. You are right that they finally consulted an Intel engineer who set them straight. The part you apparently missed is that the Intel engineer confirmed what the Intel documentation already stated. In other words, what all the Linux geniuses argued with great confidence in their mailing lists turned out to be utterly wrong. John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Mar 31 11:47:51 2002 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id C422737B41B for ; Sun, 31 Mar 2002 11:47:46 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.11.6/8.9.1) id g2VJljx89936; Sun, 31 Mar 2002 11:47:45 -0800 (PST) (envelope-from dillon) Date: Sun, 31 Mar 2002 11:47:45 -0800 (PST) From: Matthew Dillon Message-Id: <200203311947.g2VJljx89936@apollo.backplane.com> To: John Polstra Cc: smp@FreeBSD.ORG Subject: Re: RE: Syscall contention tests return, userret() bugs/issues. References: <200203311834.g2VIYrt89705@apollo.backplane.com> <200203311841.g2VIfcn18637@vashon.polstra.com> <200203311902.g2VJ28i89787@apollo.backplane.com> <200203311914.g2VJEPL18834@vashon.polstra.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org :You are not arguing against faith, you are arguing against the :documentation. I provided references for the claims I made, but I :haven't seen any references coming back the other way. Ok, I am confusing myself as much as I am confusing the list. You could very well be right in regards to main memory flushing. In fact, I think you are right. Sometimes I get on a roll and get way off track. I was thinking more of the MIPS-3 write queue and confused it with the pentium. Joy. My point is that, regardless of the Intel architecture and regardless of their documentation, it doesn't change the fact that we get big-time stalls when one cpu writes to a memory location and another cpu reads it, locked bus cycle or no locked bus cycle. It's that simple. :Bad example. I'm quite familiar with that thread. You are right that :they finally consulted an Intel engineer who set them straight. The :part you apparently missed is that the Intel engineer confirmed what :the Intel documentation already stated. In other words, what all the :Linux geniuses argued with great confidence in their mailing lists :turned out to be utterly wrong. : :John :-- : John Polstra The linux geniuses were right, actually, but they drew the wrong conclusions. What they found was a bug in earlier pentium chipsets, and they found that it was possible to stall the write pipeline indefinitely and prevent other cpu's from seeing the mutex release using carefully crafted code. -Matt To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 1 10:35: 9 2002 Delivered-To: freebsd-smp@freebsd.org Received: from one.net (ip-216-23-55-88.adsl.one.net [216.23.55.88]) by hub.freebsd.org (Postfix) with SMTP id F263337B41A for ; Mon, 1 Apr 2002 10:35:03 -0800 (PST) Received: (qmail 93264 invoked by uid 1001); 1 Apr 2002 18:35:29 -0000 Date: Mon, 1 Apr 2002 13:35:29 -0500 From: Coleman Kane To: FreeBSD-smp Subject: Interrupt delivery on SMP 760MPX board 4.5-R/4-S Message-ID: <20020401133528.A93211@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.3.22.1i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Hello, Perhaps someone here can help me. I recently purchased a Tyan Tiger MPX (S2466N) dual-athlon board, with on-board 3c920 NIC. I have been having difficulty getting multiple Tulip (DC2114x) NICs to work with this board. First, I had ordered a DLink DFE-570TX quad-channel DC21143 board to run on the server. Using both the dc and de drivers, I have beeing experiencing issues where the device doesn't appear to be delivering interrupts properly. The card does work perfectly fine on my UP EPoX 8K7A+ (AMD-760) motherboard at home. Thinking that it was perhaps a compatibility issue between the board and the bridged card (PCI-PCI DC21152 bridge), I went out and got four Asante FriendlyNET DC21140AF NICs for the system (cards I use at home). The problems persist. Anyway, I am not extremely familiar with the inner-workings of the MP spec and APIC-based hardware, and am at a loss as to what is going on. Unfortualtely, this board doesn't provide many PCI tweaking features in the BIOS, like Slot-INTPIN assignments, IRQ-PIN assignment, etc... BTW, this is running 4-stable, which I upgraded to from 4.5-R, to get the ATAPI and AMD768 support. 4.5-R experienced the bad behavior, though. I have the mptable, dmesg, and pciconf -l output if anyone could help me with this. -- coleman To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 1 13:32:45 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mailhub.yumyumyum.org (dsl092-171-091.wdc1.dsl.speakeasy.net [66.92.171.91]) by hub.freebsd.org (Postfix) with SMTP id 1E85037B41B for ; Mon, 1 Apr 2002 13:32:31 -0800 (PST) Received: (qmail 40193 invoked from network); 1 Apr 2002 21:32:01 -0000 Received: from dsl092-171-091.wdc1.dsl.speakeasy.net (66.92.171.91) by dsl092-171-091.wdc1.dsl.speakeasy.net with SMTP; 1 Apr 2002 21:32:01 -0000 Date: Mon, 1 Apr 2002 16:32:01 -0500 (EST) From: Kenneth Culver To: Coleman Kane Cc: FreeBSD-smp Subject: Re: Interrupt delivery on SMP 760MPX board 4.5-R/4-S In-Reply-To: <20020401133528.A93211@freebsd.org> Message-ID: <20020401163041.F40027-100000@alpha.yumyumyum.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org I'm not sure what the problem you're having is, but on my dual PII 333 sometime between 4.4-RELEASE and 4.5-RELEASE, my dual channel dec21140a based card stopped working properly. It was getting detected and I could ifconfig both interfaces, but only one interface would actually work. Ken On Mon, 1 Apr 2002, Coleman Kane wrote: > Hello, > Perhaps someone here can help me. I recently purchased a Tyan Tiger MPX > (S2466N) dual-athlon board, with on-board 3c920 NIC. I have been having > difficulty getting multiple Tulip (DC2114x) NICs to work with this board. > First, I had ordered a DLink DFE-570TX quad-channel DC21143 board to run > on the server. Using both the dc and de drivers, I have beeing experiencing > issues where the device doesn't appear to be delivering interrupts properly. > The card does work perfectly fine on my UP EPoX 8K7A+ (AMD-760) motherboard > at home. Thinking that it was perhaps a compatibility issue between the board > and the bridged card (PCI-PCI DC21152 bridge), I went out and got four > Asante FriendlyNET DC21140AF NICs for the system (cards I use at home). The > problems persist. Anyway, I am not extremely familiar with the inner-workings > of the MP spec and APIC-based hardware, and am at a loss as to what is going > on. Unfortualtely, this board doesn't provide many PCI tweaking features > in the BIOS, like Slot-INTPIN assignments, IRQ-PIN assignment, etc... > BTW, this is running 4-stable, which I upgraded to from 4.5-R, to get the > ATAPI and AMD768 support. 4.5-R experienced the bad behavior, though. I have > the mptable, dmesg, and pciconf -l output if anyone could help me with this. > > -- > coleman > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 1 17:30:11 2002 Delivered-To: freebsd-smp@freebsd.org Received: from k6.locore.ca (k6.locore.ca [198.96.117.170]) by hub.freebsd.org (Postfix) with ESMTP id A2C7137B417 for ; Mon, 1 Apr 2002 17:30:07 -0800 (PST) Received: (from jake@localhost) by k6.locore.ca (8.11.6/8.11.6) id g321Vif07522 for smp@freebsd.org; Mon, 1 Apr 2002 20:31:44 -0500 (EST) (envelope-from jake) Date: Mon, 1 Apr 2002 20:31:44 -0500 From: Jake Burkholder To: smp@freebsd.org Subject: ktrdump manpage Message-ID: <20020401203143.L207@locore.ca> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org ktrdump needs a manpage if anyone wants to write one. Jake To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 1 22:16:35 2002 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id 9911937B41C for ; Mon, 1 Apr 2002 22:16:31 -0800 (PST) Received: from localhost (arr@localhost) by fledge.watson.org (8.11.6/8.11.6) with SMTP id g326GRl21864 for ; Tue, 2 Apr 2002 01:16:27 -0500 (EST) (envelope-from arr@FreeBSD.org) X-Authentication-Warning: fledge.watson.org: arr owned process doing -bs Date: Tue, 2 Apr 2002 01:16:26 -0500 (EST) From: "Andrew R. Reiter" X-Sender: arr@fledge.watson.org To: smp@FreeBSD.org Subject: Where to initialize certain locks... Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org I've been looking at some global values that are not associated with any one subsystem, but need a lock at some point in order to guarantee correct bheavior -- specifically, at the moment, the securelevel value. I do not right away see a clean place for the related lock to be initialized... Am I missing some SYSINIT() (or SYSINIT()s) that are meant for helping to initialize locks in this type of situation while still protecting/promoting correct order (to ensure we init prior to a lock attempt)? Or is that not a good path to go down? Cheers, Andrew -- Andrew R. Reiter arr@watson.org arr@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 1 23:14:42 2002 Delivered-To: freebsd-smp@freebsd.org Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by hub.freebsd.org (Postfix) with ESMTP id 5246237B416; Mon, 1 Apr 2002 23:14:40 -0800 (PST) Received: by elvis.mu.org (Postfix, from userid 1192) id 0CAE8AE03F; Mon, 1 Apr 2002 23:14:40 -0800 (PST) Date: Mon, 1 Apr 2002 23:14:40 -0800 From: Alfred Perlstein To: "Andrew R. Reiter" Cc: smp@FreeBSD.org Subject: Re: Where to initialize certain locks... Message-ID: <20020402071440.GF93885@elvis.mu.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.27i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org * Andrew R. Reiter [020401 22:16] wrote: > > I've been looking at some global values that are not associated with any > one subsystem, but need a lock at some point in order to guarantee correct > bheavior -- specifically, at the moment, the securelevel value. I do not > right away see a clean place for the related lock to be initialized... Am > I missing some SYSINIT() (or SYSINIT()s) that are meant for helping to > initialize locks in this type of situation while still > protecting/promoting correct order (to ensure we init prior to a lock > attempt)? Or is that not a good path to go down? SYSINIT should work provided you run them after the mutex subsystem is setup. :) -Alfred To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 1 23:19:47 2002 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id D0BF737B416; Mon, 1 Apr 2002 23:19:42 -0800 (PST) Received: from localhost (arr@localhost) by fledge.watson.org (8.11.6/8.11.6) with SMTP id g327JWc22386; Tue, 2 Apr 2002 02:19:32 -0500 (EST) (envelope-from arr@FreeBSD.org) X-Authentication-Warning: fledge.watson.org: arr owned process doing -bs Date: Tue, 2 Apr 2002 02:19:30 -0500 (EST) From: "Andrew R. Reiter" X-Sender: arr@fledge.watson.org To: Alfred Perlstein Cc: "Andrew R. Reiter" , smp@FreeBSD.org Subject: Re: Where to initialize certain locks... In-Reply-To: <20020402071440.GF93885@elvis.mu.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Mon, 1 Apr 2002, Alfred Perlstein wrote: :* Andrew R. Reiter [020401 22:16] wrote: :> :> I've been looking at some global values that are not associated with any :> one subsystem, but need a lock at some point in order to guarantee correct :> bheavior -- specifically, at the moment, the securelevel value. I do not :> right away see a clean place for the related lock to be initialized... Am :> I missing some SYSINIT() (or SYSINIT()s) that are meant for helping to :> initialize locks in this type of situation while still :> protecting/promoting correct order (to ensure we init prior to a lock :> attempt)? Or is that not a good path to go down? : :SYSINIT should work provided you run them after the mutex subsystem is :setup. :) Well, yes :-) My main concern is that I kind of fear starting to encounter a number of these situations as we go along and need to now have a number of SYSINITs, each of which only is there to initialize a single mutex -- also the issue of where to put the SYSINITs (and the related code). Is this something Im just over thinking about? :-)) Andrew -- Andrew R. Reiter arr@watson.org arr@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 1 23:24:17 2002 Delivered-To: freebsd-smp@freebsd.org Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by hub.freebsd.org (Postfix) with ESMTP id DD7BE37B400; Mon, 1 Apr 2002 23:24:11 -0800 (PST) Received: by elvis.mu.org (Postfix, from userid 1192) id AFC90AE027; Mon, 1 Apr 2002 23:24:11 -0800 (PST) Date: Mon, 1 Apr 2002 23:24:11 -0800 From: Alfred Perlstein To: "Andrew R. Reiter" Cc: smp@FreeBSD.org Subject: Re: Where to initialize certain locks... Message-ID: <20020402072411.GH93885@elvis.mu.org> References: <20020402071440.GF93885@elvis.mu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.27i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org * Andrew R. Reiter [020401 23:19] wrote: > On Mon, 1 Apr 2002, Alfred Perlstein wrote: > > :* Andrew R. Reiter [020401 22:16] wrote: > :> > :> I've been looking at some global values that are not associated with any > :> one subsystem, but need a lock at some point in order to guarantee correct > :> bheavior -- specifically, at the moment, the securelevel value. I do not > :> right away see a clean place for the related lock to be initialized... Am > :> I missing some SYSINIT() (or SYSINIT()s) that are meant for helping to > :> initialize locks in this type of situation while still > :> protecting/promoting correct order (to ensure we init prior to a lock > :> attempt)? Or is that not a good path to go down? > : > :SYSINIT should work provided you run them after the mutex subsystem is > :setup. :) > > Well, yes :-) My main concern is that I kind of fear starting to > encounter a number of these situations as we go along and need to now have > a number of SYSINITs, each of which only is there to initialize a single > mutex -- also the issue of where to put the SYSINITs (and the related > code). Is this something Im just over thinking about? :-)) Your concern is valid, perhaps a macro that automagically makes a sysinit to init a mutex? Sort of like the "SYSCALL_MODULE_HELPER" stuff, but not as ugly? -- -Alfred Perlstein [alfred@freebsd.org] 'Instead of asking why a piece of software is using "1970s technology," start asking why software is ignoring 30 years of accumulated wisdom.' Tax deductible donations for FreeBSD: http://www.freebsdfoundation.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Apr 1 23:59:13 2002 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id 0322E37B41A; Mon, 1 Apr 2002 23:59:08 -0800 (PST) Received: from localhost (arr@localhost) by fledge.watson.org (8.11.6/8.11.6) with SMTP id g327x2g22758; Tue, 2 Apr 2002 02:59:02 -0500 (EST) (envelope-from arr@FreeBSD.org) X-Authentication-Warning: fledge.watson.org: arr owned process doing -bs Date: Tue, 2 Apr 2002 02:59:00 -0500 (EST) From: "Andrew R. Reiter" X-Sender: arr@fledge.watson.org To: Alfred Perlstein Cc: "Andrew R. Reiter" , smp@FreeBSD.org Subject: Re: Where to initialize certain locks... In-Reply-To: <20020402072411.GH93885@elvis.mu.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Mon, 1 Apr 2002, Alfred Perlstein wrote: :Your concern is valid, perhaps a macro that automagically makes :a sysinit to init a mutex? Sort of like the "SYSCALL_MODULE_HELPER" :stuff, but not as ugly? : How about something along the lines of: void mtx_sysinit(void *arg) { struct mtx_init *margs = arg; mtx_init(margs->mtx, margs->desc, margs->opts) } #define MTX_SYSINIT(name, desc, lock, opts, sub, order) \ static struct margs name##_args { \ lock, \ desc, \ opts, \ }; \ SYSINIT(name##_sysinit, sub, order, mtx_sysinit, &name##_args) -- Andrew R. Reiter arr@watson.org arr@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 0:18: 7 2002 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id 5AC1937B405; Tue, 2 Apr 2002 00:17:59 -0800 (PST) Received: from localhost (arr@localhost) by fledge.watson.org (8.11.6/8.11.6) with SMTP id g328HsS22985; Tue, 2 Apr 2002 03:17:54 -0500 (EST) (envelope-from arr@FreeBSD.org) X-Authentication-Warning: fledge.watson.org: arr owned process doing -bs Date: Tue, 2 Apr 2002 03:17:52 -0500 (EST) From: "Andrew R. Reiter" X-Sender: arr@fledge.watson.org To: "Andrew R. Reiter" Cc: Alfred Perlstein , smp@FreeBSD.org Subject: Re: Where to initialize certain locks... In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Tue, 2 Apr 2002, Andrew R. Reiter wrote: :On Mon, 1 Apr 2002, Alfred Perlstein wrote: : ::Your concern is valid, perhaps a macro that automagically makes ::a sysinit to init a mutex? Sort of like the "SYSCALL_MODULE_HELPER" ::stuff, but not as ugly? :: : :How about something along the lines of: Sorry to reply to my own post, but I feel it necesary to mention that I dislike having to pass in sub and order... : :void :mtx_sysinit(void *arg) :{ : struct mtx_init *margs = arg; : : mtx_init(margs->mtx, margs->desc, margs->opts) :} : :#define MTX_SYSINIT(name, desc, lock, opts, sub, order) \ : static struct margs name##_args { \ : lock, \ : desc, \ : opts, \ : }; \ : SYSINIT(name##_sysinit, sub, order, mtx_sysinit, &name##_args) : : :-- :Andrew R. Reiter :arr@watson.org :arr@FreeBSD.org : : -- Andrew R. Reiter arr@watson.org arr@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 0:24:46 2002 Delivered-To: freebsd-smp@freebsd.org Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by hub.freebsd.org (Postfix) with ESMTP id B925E37B41F; Tue, 2 Apr 2002 00:24:44 -0800 (PST) Received: by elvis.mu.org (Postfix, from userid 1192) id 8CADFAE026; Tue, 2 Apr 2002 00:24:44 -0800 (PST) Date: Tue, 2 Apr 2002 00:24:44 -0800 From: Alfred Perlstein To: "Andrew R. Reiter" Cc: smp@FreeBSD.org Subject: Re: Where to initialize certain locks... Message-ID: <20020402082444.GL93885@elvis.mu.org> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.3.27i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org * Andrew R. Reiter [020402 00:18] wrote: > On Tue, 2 Apr 2002, Andrew R. Reiter wrote: > > :On Mon, 1 Apr 2002, Alfred Perlstein wrote: > : > ::Your concern is valid, perhaps a macro that automagically makes > ::a sysinit to init a mutex? Sort of like the "SYSCALL_MODULE_HELPER" > ::stuff, but not as ugly? > :: > : > :How about something along the lines of: > > Sorry to reply to my own post, but I feel it necesary to mention that I > dislike having to pass in sub and order... Make it implicitly part of the macro. This code looks good so far, fix that and it should be good to go. -Alfred To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 0:38: 9 2002 Delivered-To: freebsd-smp@freebsd.org Received: from harrier.prod.itd.earthlink.net (harrier.mail.pas.earthlink.net [207.217.120.12]) by hub.freebsd.org (Postfix) with ESMTP id 4502C37B48D; Tue, 2 Apr 2002 00:37:30 -0800 (PST) Received: from pool0147.cvx22-bradley.dialup.earthlink.net ([209.179.198.147] helo=mindspring.com) by harrier.prod.itd.earthlink.net with esmtp (Exim 3.33 #1) id 16sJmm-0003pw-00; Tue, 02 Apr 2002 00:37:01 -0800 Message-ID: <3CA96D93.3DAE8C58@mindspring.com> Date: Tue, 02 Apr 2002 00:36:35 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Andrew R. Reiter" Cc: smp@FreeBSD.org Subject: Re: Where to initialize certain locks... References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org "Andrew R. Reiter" wrote: > > I've been looking at some global values that are not associated with any > one subsystem, but need a lock at some point in order to guarantee correct > bheavior -- specifically, at the moment, the securelevel value. I do not > right away see a clean place for the related lock to be initialized... Am > I missing some SYSINIT() (or SYSINIT()s) that are meant for helping to > initialize locks in this type of situation while still > protecting/promoting correct order (to ensure we init prior to a lock > attempt)? Or is that not a good path to go down? Initialized to what non-default setting? Initializing a lock as locked is a bad idea. Initializing it as unlocked is, well, automatic. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 1: 7:20 2002 Delivered-To: freebsd-smp@freebsd.org Received: from scaup.prod.itd.earthlink.net (scaup.mail.pas.earthlink.net [207.217.120.49]) by hub.freebsd.org (Postfix) with ESMTP id 3453F37B420; Tue, 2 Apr 2002 01:07:16 -0800 (PST) Received: from pool0147.cvx22-bradley.dialup.earthlink.net ([209.179.198.147] helo=mindspring.com) by scaup.prod.itd.earthlink.net with esmtp (Exim 3.33 #1) id 16sKG3-0005Qr-00; Tue, 02 Apr 2002 01:07:15 -0800 Message-ID: <3CA974A8.5440F3C9@mindspring.com> Date: Tue, 02 Apr 2002 01:06:48 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Andrew R. Reiter" Cc: Alfred Perlstein , smp@FreeBSD.org Subject: Re: Where to initialize certain locks... References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org "Andrew R. Reiter" wrote: > On Tue, 2 Apr 2002, Andrew R. Reiter wrote: > :On Mon, 1 Apr 2002, Alfred Perlstein wrote: > : > ::Your concern is valid, perhaps a macro that automagically makes > ::a sysinit to init a mutex? Sort of like the "SYSCALL_MODULE_HELPER" > ::stuff, but not as ugly? > :: > : > :How about something along the lines of: > > Sorry to reply to my own post, but I feel it necesary to mention that I > dislike having to pass in sub and order... As the original perpetrator of SYSINIT()... What would you have suggested I do instead, to ensure explicit ordering of operations, and the ability to hang operations off as "before XXX" or "after XXX", without having the full set of operations known at the time ever portion of the code was written? Realize also that the LKM facility (replaced by KLD) was intended to take the SYSINIT() sets out of modules that got loaded, and use them to initialize them, for a totally modular environment, in which the difference between a kernel module and a driver linked into the kernel was intended to be "whether or not the .so file was included in the kernel link line or not". The current stuff almost achieves this (you still can't easily distribute binary versions of drivers that need to go in the boot or module loading path), and it will likely get there eventually. The ordering has to be explicit because there is no way to know the underlying coding assumptions for any arbitrary chunk of source code, even if you have a "make depend": you don't get the toplogy information about "system call A calls subsystem B calls driver C fields interrupts with the common routine from module D onto the mbuf queue defined in F and initialized in Q, which must be called before R and S are called, or there's a panic". If you are going to complain about SYSINIT(), at least complain about something like the bubble sort in init_main.c (so I can dismiss it as a one-time operation that didn't need to be that fast, compared to, say, not doing concurrent hardware probes simultaneously for lack of a one-shot timer outcall mechanism). ;^). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 1:10:32 2002 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id 8390337B41B; Tue, 2 Apr 2002 01:10:26 -0800 (PST) Received: from localhost (arr@localhost) by fledge.watson.org (8.11.6/8.11.6) with SMTP id g329AJE23700; Tue, 2 Apr 2002 04:10:19 -0500 (EST) (envelope-from arr@FreeBSD.org) X-Authentication-Warning: fledge.watson.org: arr owned process doing -bs Date: Tue, 2 Apr 2002 04:10:18 -0500 (EST) From: "Andrew R. Reiter" X-Sender: arr@fledge.watson.org To: Terry Lambert Cc: "Andrew R. Reiter" , Alfred Perlstein , smp@FreeBSD.org Subject: Re: Where to initialize certain locks... In-Reply-To: <3CA974A8.5440F3C9@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Tue, 2 Apr 2002, Terry Lambert wrote: :If you are going to complain about SYSINIT(), at least complain :about something like the bubble sort in init_main.c (so I can :dismiss it as a one-time operation that didn't need to be that :fast, compared to, say, not doing concurrent hardware probes :simultaneously for lack of a one-shot timer outcall mechanism). Dood, you're going way off track here. I was soley disliking the fact that I had my MTX_SYSINIT() macro have the sub and order arguments. I am not and was not complaining about SYSINIT(). -- Andrew R. Reiter arr@watson.org arr@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 1:17:30 2002 Delivered-To: freebsd-smp@freebsd.org Received: from scaup.prod.itd.earthlink.net (scaup.mail.pas.earthlink.net [207.217.120.49]) by hub.freebsd.org (Postfix) with ESMTP id 06D0037B417; Tue, 2 Apr 2002 01:17:27 -0800 (PST) Received: from pool0147.cvx22-bradley.dialup.earthlink.net ([209.179.198.147] helo=mindspring.com) by scaup.prod.itd.earthlink.net with esmtp (Exim 3.33 #1) id 16sKPu-0003OM-00; Tue, 02 Apr 2002 01:17:26 -0800 Message-ID: <3CA9770B.342AEFFA@mindspring.com> Date: Tue, 02 Apr 2002 01:16:59 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "Andrew R. Reiter" Cc: Alfred Perlstein , smp@FreeBSD.org Subject: Re: Where to initialize certain locks... References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org "Andrew R. Reiter" wrote: > On Tue, 2 Apr 2002, Terry Lambert wrote: > :If you are going to complain about SYSINIT(), at least complain > :about something like the bubble sort in init_main.c (so I can > :dismiss it as a one-time operation that didn't need to be that > :fast, compared to, say, not doing concurrent hardware probes > :simultaneously for lack of a one-shot timer outcall mechanism). > > Dood, you're going way off track here. I was soley disliking the fact > that I had my MTX_SYSINIT() macro have the sub and order arguments. I am > not and was not complaining about SYSINIT(). Oh. That's very different, then. Never mind. 8-) 8-). For that, you can just make your MTX_SYSINIT() not take args, and pass in SI_SUB_LOCK and SI_ORDER_MIDDLE. You probably don't want to do it at SI_SUB_MTX_POOL, unless you can guarantee there will be no allocations (someone else should be able to tell you if this is a valid assumption for a mutex, and whether it's likely to remain so for all time). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 8:10:13 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail11.speakeasy.net (mail11.speakeasy.net [216.254.0.211]) by hub.freebsd.org (Postfix) with ESMTP id 74FB437B49D for ; Tue, 2 Apr 2002 08:09:10 -0800 (PST) Received: (qmail 15510 invoked from network); 2 Apr 2002 16:09:09 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail11.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 2 Apr 2002 16:09:09 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g32G9tv19380; Tue, 2 Apr 2002 11:09:55 -0500 (EST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3CA96D93.3DAE8C58@mindspring.com> Date: Tue, 02 Apr 2002 11:09:10 -0500 (EST) From: John Baldwin To: Terry Lambert Subject: Re: Where to initialize certain locks... Cc: smp@FreeBSD.org, "Andrew R. Reiter" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 02-Apr-2002 Terry Lambert wrote: > "Andrew R. Reiter" wrote: >> >> I've been looking at some global values that are not associated with any >> one subsystem, but need a lock at some point in order to guarantee correct >> bheavior -- specifically, at the moment, the securelevel value. I do not >> right away see a clean place for the related lock to be initialized... Am >> I missing some SYSINIT() (or SYSINIT()s) that are meant for helping to >> initialize locks in this type of situation while still >> protecting/promoting correct order (to ensure we init prior to a lock >> attempt)? Or is that not a good path to go down? > > Initialized to what non-default setting? > > Initializing a lock as locked is a bad idea. Initializing it as > unlocked is, well, automatic. No it's not. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 11:55:43 2002 Delivered-To: freebsd-smp@freebsd.org Received: from one.net (ip-216-23-55-88.adsl.one.net [216.23.55.88]) by hub.freebsd.org (Postfix) with SMTP id E3B3137B417 for ; Tue, 2 Apr 2002 11:55:13 -0800 (PST) Received: (qmail 14884 invoked by uid 1001); 2 Apr 2002 19:55:40 -0000 Date: Tue, 2 Apr 2002 14:55:40 -0500 From: Coleman Kane To: Kenneth Culver Cc: FreeBSD-smp , FreeBSD-stable , Chris Subject: Re: Interrupt delivery on SMP 760MPX board 4.5-R/4-S Message-ID: <20020402145540.A14822@freebsd.org> References: <20020401133528.A93211@freebsd.org> <20020401163041.F40027-100000@alpha.yumyumyum.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="huq684BweRXVnRxX" Content-Disposition: inline In-Reply-To: <20020401163041.F40027-100000@alpha.yumyumyum.org> User-Agent: Mutt/1.3.22.1i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org --huq684BweRXVnRxX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I believe it has something to do with the APIC on this board. I have a production Dual PIII VIA server running 4.2 to compare to and I noticed a few things: 1) This Dual Athlon machine is still routing 16 INT's in ICU-fashion 2) The ExtINT pin on the PIII system, in local INTs, is routed via a PCI bus, but the Athlon System routs this via the ISA bus. I think this is wrong.=20 3) Furthermore, all INTs are routed in ISA-fashion and the Bus in mptable is ISA (except NMI). There are no PCI routing pins listed at all (A-D). 4) The 5.0-CURRENT dmesg/mptable dump from the GUINNESS reference board also displays this behavior, which makes me think that it perhaps doesn't completely work, a bridged PCI card like my DFE-570TX might be worth a t= ry. Ideally, from what I can see, PCI devices that are attached via=20 IntA,B,C,D,etc.. and handled by the I/O APIC instead of the 8259's should be running off irq's > 15. They should also have PCI bus attachments for INTs listed in mptable. This is probably worth looking into, I grazed over the MP 1.4 spec, as well as some AMD tech docs. It would appear that for some reason, the INT is routed, but the OS doesn't set up the APIC properly, or maybe this is expected to be the job of the BIOS... Either way, it=20 seems as if the Dual Athlon boards (at least ones I've seen) do not fully work yet. =20 -- coleman kane On Mon, Apr 01, 2002 at 04:32:01PM -0500, Kenneth Culver wrote: > I'm not sure what the problem you're having is, but on my dual PII 333 > sometime between 4.4-RELEASE and 4.5-RELEASE, my dual channel dec21140a > based card stopped working properly. It was getting detected and I could > ifconfig both interfaces, but only one interface would actually work. >=20 > Ken >=20 > On Mon, 1 Apr 2002, Coleman Kane wrote: >=20 > > Hello, > > Perhaps someone here can help me. I recently purchased a Tyan Tiger MPX > > (S2466N) dual-athlon board, with on-board 3c920 NIC. I have been having > > difficulty getting multiple Tulip (DC2114x) NICs to work with this boar= d. > > First, I had ordered a DLink DFE-570TX quad-channel DC21143 board to run > > on the server. Using both the dc and de drivers, I have beeing experien= cing > > issues where the device doesn't appear to be delivering interrupts prop= erly. > > The card does work perfectly fine on my UP EPoX 8K7A+ (AMD-760) motherb= oard > > at home. Thinking that it was perhaps a compatibility issue between the= board > > and the bridged card (PCI-PCI DC21152 bridge), I went out and got four > > Asante FriendlyNET DC21140AF NICs for the system (cards I use at home).= The > > problems persist. Anyway, I am not extremely familiar with the inner-wo= rkings > > of the MP spec and APIC-based hardware, and am at a loss as to what is = going > > on. Unfortualtely, this board doesn't provide many PCI tweaking features > > in the BIOS, like Slot-INTPIN assignments, IRQ-PIN assignment, etc... > > BTW, this is running 4-stable, which I upgraded to from 4.5-R, to get t= he > > ATAPI and AMD768 support. 4.5-R experienced the bad behavior, though. I= have > > the mptable, dmesg, and pciconf -l output if anyone could help me with = this. > > > > -- > > coleman > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > > with "unsubscribe freebsd-smp" in the body of the message > > > > >=20 --huq684BweRXVnRxX Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (FreeBSD) iD8DBQE8qgy7ERViMObJ880RApuBAJ4pT7lfeqa5sHKSbHIpbpE76GXIjQCfYrjH 6VJRubCA2iemkQ4tslQ+RBM= =Boic -----END PGP SIGNATURE----- --huq684BweRXVnRxX-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 13: 4:14 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mailhub.yumyumyum.org (dsl092-171-091.wdc1.dsl.speakeasy.net [66.92.171.91]) by hub.freebsd.org (Postfix) with SMTP id 5DE9337B447 for ; Tue, 2 Apr 2002 13:03:55 -0800 (PST) Received: (qmail 47663 invoked from network); 2 Apr 2002 21:03:20 -0000 Received: from dsl092-171-091.wdc1.dsl.speakeasy.net (66.92.171.91) by dsl092-171-091.wdc1.dsl.speakeasy.net with SMTP; 2 Apr 2002 21:03:20 -0000 Date: Tue, 2 Apr 2002 16:03:20 -0500 (EST) From: Kenneth Culver To: Coleman Kane Cc: FreeBSD-smp , FreeBSD-stable , Chris Subject: Re: Interrupt delivery on SMP 760MPX board 4.5-R/4-S In-Reply-To: <20020402145540.A14822@freebsd.org> Message-ID: <20020402160203.K47643-100000@alpha.yumyumyum.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > I believe it has something to do with the APIC on this board. I have a > production Dual PIII VIA server running 4.2 to compare to and I noticed a > few things: > 1) This Dual Athlon machine is still routing 16 INT's in ICU-fashion > 2) The ExtINT pin on the PIII system, in local INTs, is routed via a > PCI bus, but the Athlon System routs this via the ISA bus. I think > this is wrong. > 3) Furthermore, all INTs are routed in ISA-fashion and the Bus in mptable > is ISA (except NMI). There are no PCI routing pins listed at all (A-D). > 4) The 5.0-CURRENT dmesg/mptable dump from the GUINNESS reference board > also displays this behavior, which makes me think that it perhaps doesn't > completely work, a bridged PCI card like my DFE-570TX might be worth a try. > > Ideally, from what I can see, PCI devices that are attached via > IntA,B,C,D,etc.. and handled by the I/O APIC instead of the 8259's should > be running off irq's > 15. They should also have PCI bus attachments for > INTs listed in mptable. This is probably worth looking into, I grazed over > the MP 1.4 spec, as well as some AMD tech docs. It would appear that for > some reason, the INT is routed, but the OS doesn't set up the APIC properly, > or maybe this is expected to be the job of the BIOS... Either way, it > seems as if the Dual Athlon boards (at least ones I've seen) do not fully > work yet. > Even with all that said, my Dual Proc PII 333, based on an intel mobo, does not work right with FreeBSD 4.5, and broke in between 4.4 and 4.5. It works but does not support the dual channel card. I don't know what caused this but I do know this is the behavior I'm seeing. :-) Ken To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 13:39:54 2002 Delivered-To: freebsd-smp@freebsd.org Received: from magnesium.cs.washington.edu (magnesium.cs.washington.edu [128.95.3.94]) by hub.freebsd.org (Postfix) with ESMTP id EA67D37B400 for ; Tue, 2 Apr 2002 13:39:17 -0800 (PST) Received: from localhost (tzoompy@localhost) by magnesium.cs.washington.edu (8.11.6/8.11.6/0.4) with ESMTP id g32LdHV28754 for ; Tue, 2 Apr 2002 13:39:17 -0800 X-Authentication-Warning: magnesium.cs.washington.edu: tzoompy owned process doing -bs Date: Tue, 2 Apr 2002 13:39:17 -0800 (PST) From: Stefan Saroiu To: freebsd-smp@FreeBSD.org Subject: Interrupt Handlers and Multiple CPUs Message-ID: <20020402132930.B28718-100000@magnesium.cs.washington.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Hello, This is the context. I am writing a packet capture application on a PIV 2GHz single proc machine with a SK-9843 1Gbps network card. Under 6-700 Mbps of traffic, 60-70% of the CPU is spent inside of the interrupt handler. This is the interrupt coming from the NIC (pci/if_sk.c). Because so many CPU cycles are spent on these interrupts, the packet capture application gets starved over time. Now the question: ================ If I drop a second CPU in the box, will FreeBSD be smart to schedule the interrupts on one CPU and the packet capture code on the other? What FreeBSD version do I need (I have 4.5)? The SMP docs on the FreeBSD site seem to point to an affirmative answer, but I'd like to get more opinions before I blow more money on a second CPU. Many thanks, --Stefan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 13:47:49 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mailhub.yumyumyum.org (dsl092-171-091.wdc1.dsl.speakeasy.net [66.92.171.91]) by hub.freebsd.org (Postfix) with SMTP id 0CE6337B41A for ; Tue, 2 Apr 2002 13:47:46 -0800 (PST) Received: (qmail 47996 invoked from network); 2 Apr 2002 21:47:11 -0000 Received: from dsl092-171-091.wdc1.dsl.speakeasy.net (66.92.171.91) by dsl092-171-091.wdc1.dsl.speakeasy.net with SMTP; 2 Apr 2002 21:47:11 -0000 Date: Tue, 2 Apr 2002 16:47:11 -0500 (EST) From: Kenneth Culver To: Stefan Saroiu Cc: freebsd-smp@FreeBSD.ORG Subject: Re: Interrupt Handlers and Multiple CPUs In-Reply-To: <20020402132930.B28718-100000@magnesium.cs.washington.edu> Message-ID: <20020402164407.L47960-100000@alpha.yumyumyum.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > Hello, > > This is the context. > > I am writing a packet capture application on a PIV 2GHz single proc > machine with a SK-9843 1Gbps network card. Under 6-700 Mbps of traffic, > 60-70% of the CPU is spent inside of the interrupt handler. This > is the interrupt coming from the NIC (pci/if_sk.c). > > Because so many CPU cycles are spent on these interrupts, the packet > capture application gets starved over time. > > Now the question: > ================ > > If I drop a second CPU in the box, will FreeBSD be smart to schedule the > interrupts on one CPU and the packet capture code on the other? What > FreeBSD version do I need (I have 4.5)? > > The SMP docs on the FreeBSD site seem to point to an affirmative answer, > but I'd like to get more opinions before I blow more money on a second > CPU. > I'm assuming the packet capture code is in userland, and if that's the case, then the answer is "the interrupt handler and the packet cap code may or may not run on the same cpu, meaning they might run on seperate cpu's. This is FreeBSD-4.x. So the answer I guess is maybe but it's not guarunteed to do it all the time, and as far as I know, there's no way to force this behavior. Ken To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 15:44: 6 2002 Delivered-To: freebsd-smp@freebsd.org Received: from falcon.prod.itd.earthlink.net (falcon.mail.pas.earthlink.net [207.217.120.74]) by hub.freebsd.org (Postfix) with ESMTP id AD34A37B416; Tue, 2 Apr 2002 15:44:02 -0800 (PST) Received: from pool0045.cvx22-bradley.dialup.earthlink.net ([209.179.198.45] helo=mindspring.com) by falcon.prod.itd.earthlink.net with esmtp (Exim 3.33 #1) id 16sXwX-0003k5-00; Tue, 02 Apr 2002 15:44:01 -0800 Message-ID: <3CAA422A.4CAF1CAE@mindspring.com> Date: Tue, 02 Apr 2002 15:43:38 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: smp@FreeBSD.org, "Andrew R. Reiter" Subject: Re: Where to initialize certain locks... References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > > Initialized to what non-default setting? > > > > Initializing a lock as locked is a bad idea. Initializing it as > > unlocked is, well, automatic. > > No it's not. Then the design is broken. It means you can't statically declare locks. This is one of the reasons the POSIX mutexes sucked in Draft 4, and one of the major reasons the standard version of pthreads was better than draft 4. If this is the case, then there needs to be constructor-intialization available (see the Moscow Center for Supercomputing Activities support for static class declaration of mutex containing classes, for NT and Draft 4 POSIX threads -- I wrote the Draft 4 support). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 16:16:30 2002 Delivered-To: freebsd-smp@freebsd.org Received: from one.net (ip-216-23-55-88.adsl.one.net [216.23.55.88]) by hub.freebsd.org (Postfix) with SMTP id 4D4A437B400 for ; Tue, 2 Apr 2002 16:16:24 -0800 (PST) Received: (qmail 40943 invoked by uid 1001); 3 Apr 2002 00:16:48 -0000 Date: Tue, 2 Apr 2002 19:16:48 -0500 From: Coleman Kane To: Kenneth Culver Cc: FreeBSD-smp , FreeBSD-stable , Chris Subject: Re: Interrupt delivery on SMP 760MPX board 4.5-R/4-S Message-ID: <20020402191648.A40926@freebsd.org> References: <20020402145540.A14822@freebsd.org> <20020402160203.K47643-100000@alpha.yumyumyum.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20020402160203.K47643-100000@alpha.yumyumyum.org> User-Agent: Mutt/1.3.22.1i Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Hey, send me over a dmesg+mptable output, your problem may be something else. -- coleman On Tue, Apr 02, 2002 at 04:03:20PM -0500, Kenneth Culver wrote: > > I believe it has something to do with the APIC on this board. I have a > > production Dual PIII VIA server running 4.2 to compare to and I noticed a > > few things: > > 1) This Dual Athlon machine is still routing 16 INT's in ICU-fashion > > 2) The ExtINT pin on the PIII system, in local INTs, is routed via a > > PCI bus, but the Athlon System routs this via the ISA bus. I think > > this is wrong. > > 3) Furthermore, all INTs are routed in ISA-fashion and the Bus in mptable > > is ISA (except NMI). There are no PCI routing pins listed at all (A-D). > > 4) The 5.0-CURRENT dmesg/mptable dump from the GUINNESS reference board > > also displays this behavior, which makes me think that it perhaps doesn't > > completely work, a bridged PCI card like my DFE-570TX might be worth a try. > > > > Ideally, from what I can see, PCI devices that are attached via > > IntA,B,C,D,etc.. and handled by the I/O APIC instead of the 8259's should > > be running off irq's > 15. They should also have PCI bus attachments for > > INTs listed in mptable. This is probably worth looking into, I grazed over > > the MP 1.4 spec, as well as some AMD tech docs. It would appear that for > > some reason, the INT is routed, but the OS doesn't set up the APIC properly, > > or maybe this is expected to be the job of the BIOS... Either way, it > > seems as if the Dual Athlon boards (at least ones I've seen) do not fully > > work yet. > > > Even with all that said, my Dual Proc PII 333, based on an intel mobo, > does not work right with FreeBSD 4.5, and broke in between 4.4 and 4.5. It > works but does not support the dual channel card. I don't know what caused > this but I do know this is the behavior I'm seeing. :-) > > Ken > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 16:20: 6 2002 Delivered-To: freebsd-smp@freebsd.org Received: from k6.locore.ca (k6.locore.ca [198.96.117.170]) by hub.freebsd.org (Postfix) with ESMTP id 233AE37B41C; Tue, 2 Apr 2002 16:19:51 -0800 (PST) Received: (from jake@localhost) by k6.locore.ca (8.11.6/8.11.6) id g330Ljs13918; Tue, 2 Apr 2002 19:21:45 -0500 (EST) (envelope-from jake) Date: Tue, 2 Apr 2002 19:21:44 -0500 From: Jake Burkholder To: Terry Lambert Cc: John Baldwin , smp@FreeBSD.ORG, "Andrew R. Reiter" Subject: Re: Where to initialize certain locks... Message-ID: <20020402192144.Q207@locore.ca> References: <3CAA422A.4CAF1CAE@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3CAA422A.4CAF1CAE@mindspring.com>; from tlambert2@mindspring.com on Tue, Apr 02, 2002 at 03:43:38PM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Apparently, On Tue, Apr 02, 2002 at 03:43:38PM -0800, Terry Lambert said words to the effect of; > John Baldwin wrote: > > > Initialized to what non-default setting? > > > > > > Initializing a lock as locked is a bad idea. Initializing it as > > > unlocked is, well, automatic. > > > > No it's not. > > Then the design is broken. It means you can't statically declare > locks. This is one of the reasons the POSIX mutexes sucked in > Draft 4, and one of the major reasons the standard version of > pthreads was better than draft 4. > > If this is the case, then there needs to be constructor-intialization > available (see the Moscow Center for Supercomputing Activities > support for static class declaration of mutex containing classes, > for NT and Draft 4 POSIX threads -- I wrote the Draft 4 support). The reason they need to be initialized is to insert them into an all_mtx list which is used for debugging. Otherwise there's no problem; the lock for said list is statically initialized. This is the kernel. This is not pthreads. Arguably anything that depends on them needing to be statically initialized to unlocked and usable is broken. Jake To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 16:48:22 2002 Delivered-To: freebsd-smp@freebsd.org Received: from hawk.mail.pas.earthlink.net (hawk.mail.pas.earthlink.net [207.217.120.22]) by hub.freebsd.org (Postfix) with ESMTP id 0DDF037B41E for ; Tue, 2 Apr 2002 16:48:07 -0800 (PST) Received: from pool0116.cvx40-bradley.dialup.earthlink.net ([216.244.42.116] helo=mindspring.com) by hawk.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 16sYwT-0002J0-00; Tue, 02 Apr 2002 16:48:01 -0800 Message-ID: <3CAA512A.9FBEAE5@mindspring.com> Date: Tue, 02 Apr 2002 16:47:38 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Stefan Saroiu Cc: freebsd-smp@FreeBSD.org Subject: Re: Interrupt Handlers and Multiple CPUs References: <20020402132930.B28718-100000@magnesium.cs.washington.edu> Content-Type: text/plain; charset=EUC-KR Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Stefan Saroiu wrote: > This is the context. > > I am writing a packet capture application on a PIV 2GHz single proc > machine with a SK-9843 1Gbps network card. Under 6-700 Mbps of traffic, > 60-70% of the CPU is spent inside of the interrupt handler. This > is the interrupt coming from the NIC (pci/if_sk.c). > > Because so many CPU cycles are spent on these interrupts, the packet > capture application gets starved over time. This is called "receiver livelock". Read the DEC Western Research Laboratories paper by Jaff Mogul on this from 1991, then read the Rice University papers on the from 1196 and 1998 by Peter Druschel and Mohit Aron. > Now the question: > ================ > > If I drop a second CPU in the box, will FreeBSD be smart to schedule the > interrupts on one CPU and the packet capture code on the other? What > FreeBSD version do I need (I have 4.5)? > > The SMP docs on the FreeBSD site seem to point to an affirmative answer, > but I'd like to get more opinions before I blow more money on a second > CPU. [ I assume that this is for a commercial "1 Gigbit!" capture tool... ] The interrupts will be scheduled on the CPU that has the vector. In the common case, this will be the same CPU that's running the capture application, because it's "in the kernel" waiting for I/O from the card, and thereore has the kernel reentrancy lock, which implicitly gives it the vector for the interrupts. Ideally, FreeBSD would have two other modes of operating: 1) Direct vector of interrupts, in which interrupts from a specific card are directed to a particular CPU, AND 2) Virtual wire mode, in which interrupts are directed to a "virtual" interrupt line, and whatever CPU wants to handle them can. The main barrier to both of these approaches is that only one CPU is allowed into the kernel code at a time in FreeBSD 4.x, and the entry barrier points are "system call", "fault handler", and "interrupt handler". Druschel and Aron have implemented a fix for the problem you are seeing, called LRP ("Lazy Receiver Processing"). This has been available for FreeBSD since the 2.x days, and you can download code for FreeBSD 3.x and 4.x. The 4.x code should work for you. It contains accounting constructis for resource accounting, to ensure scheduling of user space processes, thus avoiding the live lock. This implementation is called ResCon ("Resource Containers"). The license on this code prohibits commercial use without contacting the Rice University Technology Transfer office, but it should be enough to let you see what you are missing, and whether or not you want to license. In general, I would say that throwing hardware at the problem will not work, and any company that thinks that this is a workable solution, regardless of what their platform is, will find themselves disappointed. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 17:19:37 2002 Delivered-To: freebsd-smp@freebsd.org Received: from hawk.mail.pas.earthlink.net (hawk.mail.pas.earthlink.net [207.217.120.22]) by hub.freebsd.org (Postfix) with ESMTP id AB7C637B419; Tue, 2 Apr 2002 17:19:31 -0800 (PST) Received: from pool0116.cvx40-bradley.dialup.earthlink.net ([216.244.42.116] helo=mindspring.com) by hawk.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 16sZQv-0004ia-00; Tue, 02 Apr 2002 17:19:29 -0800 Message-ID: <3CAA5888.11E9406E@mindspring.com> Date: Tue, 02 Apr 2002 17:19:04 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Jake Burkholder Cc: John Baldwin , smp@FreeBSD.ORG, "Andrew R. Reiter" Subject: Re: Where to initialize certain locks... References: <3CAA422A.4CAF1CAE@mindspring.com> <20020402192144.Q207@locore.ca> Content-Type: text/plain; charset=EUC-KR Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Jake Burkholder wrote: > The reason they need to be initialized is to insert them into an all_mtx > list which is used for debugging. Otherwise there's no problem; the lock > for said list is statically initialized. > > This is the kernel. This is not pthreads. Arguably anything that depends > on them needing to be statically initialized to unlocked and usable is > broken. So it should be "#ifdef DEBUG". In the non-debug case, this should be done using linker sets. Here is some sample code: struct foo { int some_data; struct foo *next; }; #define FOOINIT(uniquifier, data) \ static struct foo uniquifier ## _foo_init = { \ data, \ 0 \ }; \ DATA_SET(foo_set,uniquifier ## _foo_init); void foo_startup( void) { register struct foo **foopp; /* * Turn sysinit list into linked list so we can * extend it later by linking things onto the * next pointer. */ for( foopp = foo; *foopp != NULL; foopp++) { if ( *(foopp + 1) != NULL) (*foopp)->next = *(foopp + 1); } } Voila'. You now have a linked list, like that used for debugging, built out of linker set entries, and the list is runtime extensible, unlike, say, the VOP descriptor list. You can now statically declare mutex instances, and they will be immediately ready for use, after the statup that does the linkage runs. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Apr 2 17:49:50 2002 Delivered-To: freebsd-smp@freebsd.org Received: from magnesium.cs.washington.edu (magnesium.cs.washington.edu [128.95.3.94]) by hub.freebsd.org (Postfix) with ESMTP id 2BACB37B41C for ; Tue, 2 Apr 2002 17:49:45 -0800 (PST) Received: from localhost (tzoompy@localhost) by magnesium.cs.washington.edu (8.11.6/8.11.6/0.4) with ESMTP id g331nhU29166; Tue, 2 Apr 2002 17:49:43 -0800 X-Authentication-Warning: magnesium.cs.washington.edu: tzoompy owned process doing -bs Date: Tue, 2 Apr 2002 17:49:43 -0800 (PST) From: Stefan Saroiu To: Terry Lambert Cc: freebsd-smp@FreeBSD.org Subject: Re: Interrupt Handlers and Multiple CPUs In-Reply-To: <3CAA512A.9FBEAE5@mindspring.com> Message-ID: <20020402174143.B29152-100000@magnesium.cs.washington.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Tue, 2 Apr 2002, Terry Lambert wrote: > Stefan Saroiu wrote: > > This is the context. > > > > Now the question: > > ================ > > > > If I drop a second CPU in the box, will FreeBSD be smart to schedule the > > interrupts on one CPU and the packet capture code on the other? What > > FreeBSD version do I need (I have 4.5)? > > > > The SMP docs on the FreeBSD site seem to point to an affirmative answer, > > but I'd like to get more opinions before I blow more money on a second > > CPU. > > [ I assume that this is for a commercial "1 Gigbit!" capture tool... ] > On the contrary, this is for an academic project. I'm a grad student. > The interrupts will be scheduled on the CPU that has the vector. > > In the common case, this will be the same CPU that's running the > capture application, because it's "in the kernel" waiting for I/O > from the card, and thereore has the kernel reentrancy lock, which > implicitly gives it the vector for the interrupts. > > Ideally, FreeBSD would have two other modes of operating: > > 1) Direct vector of interrupts, in which interrupts from > a specific card are directed to a particular CPU, AND > > 2) Virtual wire mode, in which interrupts are directed > to a "virtual" interrupt line, and whatever CPU wants > to handle them can. > > The main barrier to both of these approaches is that only one > CPU is allowed into the kernel code at a time in FreeBSD 4.x, > and the entry barrier points are "system call", "fault handler", > and "interrupt handler". > Your answer seems to (indirectly) suggest that I might have a chance with 5.0, since they do more fine grained kernel locking. Any suggestions? > Druschel and Aron have implemented a fix for the problem you > are seeing, called LRP ("Lazy Receiver Processing"). This has > been available for FreeBSD since the 2.x days, and you can > download code for FreeBSD 3.x and 4.x. The 4.x code should work > for you. It contains accounting constructis for resource > accounting, to ensure scheduling of user space processes, thus > avoiding the live lock. This implementation is called ResCon > ("Resource Containers"). > The application still gets 20% of the CPU which is quite good actually. Although I'm not familiar with Druschel's work, I'm not sure whether better scheduling will help me here. I've been toying with the idea to change the driver to raise interrupts only once every 100 packets or something like that. Currently it is 1 interrupt per 1 packet. > The license on this code prohibits commercial use without > contacting the Rice University Technology Transfer office, > but it should be enough to let you see what you are missing, > and whether or not you want to license. > Again, this is purely academic, I shouldn't have a problem. > > In general, I would say that throwing hardware at the problem > will not work, and any company that thinks that this is a > workable solution, regardless of what their platform is, will > find themselves disappointed. > Thanks for your answer. It's very informative. Cheers, --Stefan > -- Terry > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 3 7:12: 3 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail13.speakeasy.net (mail13.speakeasy.net [216.254.0.213]) by hub.freebsd.org (Postfix) with ESMTP id 26E3537B419 for ; Wed, 3 Apr 2002 07:11:53 -0800 (PST) Received: (qmail 19472 invoked from network); 3 Apr 2002 15:11:51 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail13.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 3 Apr 2002 15:11:51 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g33FCav23425; Wed, 3 Apr 2002 10:12:36 -0500 (EST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3CAA5888.11E9406E@mindspring.com> Date: Wed, 03 Apr 2002 10:11:53 -0500 (EST) From: John Baldwin To: Terry Lambert Subject: Re: Where to initialize certain locks... Cc: "Andrew R. Reiter" , smp@FreeBSD.ORG, Jake Burkholder Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 03-Apr-2002 Terry Lambert wrote: > Jake Burkholder wrote: >> The reason they need to be initialized is to insert them into an all_mtx >> list which is used for debugging. Otherwise there's no problem; the lock >> for said list is statically initialized. >> >> This is the kernel. This is not pthreads. Arguably anything that depends >> on them needing to be statically initialized to unlocked and usable is >> broken. > > So it should be "#ifdef DEBUG". > > In the non-debug case, this should be done using linker sets. > Here is some sample code: > > struct foo { > int some_data; > struct foo *next; > }; > > #define FOOINIT(uniquifier, data) \ > static struct foo uniquifier ## _foo_init = { \ > data, \ > 0 \ > }; \ > DATA_SET(foo_set,uniquifier ## _foo_init); > > void > foo_startup( void) > { > register struct foo **foopp; > > /* > * Turn sysinit list into linked list so we can > * extend it later by linking things onto the > * next pointer. > */ > for( foopp = foo; *foopp != NULL; foopp++) { > if ( *(foopp + 1) != NULL) > (*foopp)->next = *(foopp + 1); > } > } > > Voila'. You now have a linked list, like that used for debugging, > built out of linker set entries, and the list is runtime extensible, > unlike, say, the VOP descriptor list. > > You can now statically declare mutex instances, and they will be > immediately ready for use, after the statup that does the linkage > runs. We need mutexes prior to mi_startup(). -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 3 8:16:32 2002 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id 4A34D37B41B for ; Wed, 3 Apr 2002 08:16:24 -0800 (PST) Received: from fledge.watson.org (fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.11.6/8.11.6) with SMTP id g33GG7w48462; Wed, 3 Apr 2002 11:16:07 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Wed, 3 Apr 2002 11:16:07 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Stefan Saroiu Cc: Terry Lambert , freebsd-smp@FreeBSD.org Subject: Re: Interrupt Handlers and Multiple CPUs In-Reply-To: <20020402174143.B29152-100000@magnesium.cs.washington.edu> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Tue, 2 Apr 2002, Stefan Saroiu wrote: > The application still gets 20% of the CPU which is quite good actually. > Although I'm not familiar with Druschel's work, I'm not sure whether > better scheduling will help me here. > > I've been toying with the idea to change the driver to raise interrupts > only once every 100 packets or something like that. Currently it is 1 > interrupt per 1 packet. Ouch. No wonder you're having problems. You definitely want to implement one of coalesced interrupt delivery or polled device access. In theory, we have both in 4.x and 5.x, but support for coalesced delivery is on a per-card basis. 5.x will allow you do to the kinds of things you want (eventually) once the network stack is fine-grained enough, but it sounds like the big problem is the driver model. I believe fxp and em drivers support this, and might be a good model to look at. You might want to consider posting to freebsd-net for pointers in this space. Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 3 12:12:38 2002 Delivered-To: freebsd-smp@freebsd.org Received: from swan.prod.itd.earthlink.net (swan.mail.pas.earthlink.net [207.217.120.123]) by hub.freebsd.org (Postfix) with ESMTP id 21FF437B41A; Wed, 3 Apr 2002 12:12:35 -0800 (PST) Received: from pool0355.cvx40-bradley.dialup.earthlink.net ([216.244.43.100] helo=mindspring.com) by swan.prod.itd.earthlink.net with esmtp (Exim 3.33 #1) id 16sr7S-0005ZD-00; Wed, 03 Apr 2002 12:12:34 -0800 Message-ID: <3CAB621A.6F84F736@mindspring.com> Date: Wed, 03 Apr 2002 12:12:10 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: "Andrew R. Reiter" , smp@FreeBSD.ORG, Jake Burkholder Subject: Re: Where to initialize certain locks... References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > We need mutexes prior to mi_startup(). There are really two problems with this theory: 1) You can't use them before the mutex system is initialized, which happens from mi_startup. 2) There are no contended resources until after mi_startup starts the other CPU(s). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 3 12:14:19 2002 Delivered-To: freebsd-smp@freebsd.org Received: from swan.prod.itd.earthlink.net (swan.mail.pas.earthlink.net [207.217.120.123]) by hub.freebsd.org (Postfix) with ESMTP id 4E44E37B41A; Wed, 3 Apr 2002 12:14:15 -0800 (PST) Received: from pool0355.cvx40-bradley.dialup.earthlink.net ([216.244.43.100] helo=mindspring.com) by swan.prod.itd.earthlink.net with esmtp (Exim 3.33 #1) id 16sr94-0000U2-00; Wed, 03 Apr 2002 12:14:14 -0800 Message-ID: <3CAB627F.79EF279@mindspring.com> Date: Wed, 03 Apr 2002 12:13:51 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin , "Andrew R. Reiter" , smp@FreeBSD.ORG, Jake Burkholder Subject: Re: Where to initialize certain locks... References: <3CAB621A.6F84F736@mindspring.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Terry Lambert wrote: > John Baldwin wrote: > > We need mutexes prior to mi_startup(). > > There are really two problems with this theory: > > 1) You can't use them before the mutex system is initialized, > which happens from mi_startup. > > 2) There are no contended resources until after mi_startup > starts the other CPU(s). Forgot: 3) I never said to call the init routine at mi_startup(); you could actually call it as early as you wanted, as long as there were no dependencies on things like memory allocation. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 3 12:29:18 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mail11.speakeasy.net (mail11.speakeasy.net [216.254.0.211]) by hub.freebsd.org (Postfix) with ESMTP id 0B02137B42F for ; Wed, 3 Apr 2002 12:29:05 -0800 (PST) Received: (qmail 22883 invoked from network); 3 Apr 2002 20:29:03 -0000 Received: from unknown (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) by mail11.speakeasy.net (qmail-ldap-1.03) with DES-CBC3-SHA encrypted SMTP for ; 3 Apr 2002 20:29:03 -0000 Received: from laptop.baldwin.cx (gw1.twc.weather.com [216.133.140.1]) by server.baldwin.cx (8.11.6/8.11.6) with ESMTP id g33KTmv24554; Wed, 3 Apr 2002 15:29:48 -0500 (EST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.5.2 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3CAB621A.6F84F736@mindspring.com> Date: Wed, 03 Apr 2002 15:29:04 -0500 (EST) From: John Baldwin To: Terry Lambert Subject: Re: Where to initialize certain locks... Cc: Jake Burkholder , smp@FreeBSD.ORG, "Andrew R. Reiter" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 03-Apr-2002 Terry Lambert wrote: > John Baldwin wrote: >> We need mutexes prior to mi_startup(). > > There are really two problems with this theory: > > 1) You can't use them before the mutex system is initialized, > which happens from mi_startup. Incorrect. It happens prior to mi_startup(). > 2) There are no contended resources until after mi_startup > starts the other CPU(s). But there is lots of code that we don't want to have to do stuff like: if (!cold) mtx_lock(&foo_mtx); else /* just wing it */ Easier to just always acquire the locks and init them early enough. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 3 12:35:20 2002 Delivered-To: freebsd-smp@freebsd.org Received: from hawk.mail.pas.earthlink.net (hawk.mail.pas.earthlink.net [207.217.120.22]) by hub.freebsd.org (Postfix) with ESMTP id 1016C37B416; Wed, 3 Apr 2002 12:34:47 -0800 (PST) Received: from pool0355.cvx40-bradley.dialup.earthlink.net ([216.244.43.100] helo=mindspring.com) by hawk.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 16srSs-0000li-00; Wed, 03 Apr 2002 12:34:42 -0800 Message-ID: <3CAB674A.F6BC6F9D@mindspring.com> Date: Wed, 03 Apr 2002 12:34:18 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: Robert Watson Cc: Stefan Saroiu , freebsd-smp@FreeBSD.org Subject: Re: Interrupt Handlers and Multiple CPUs References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Robert Watson wrote: > On Tue, 2 Apr 2002, Stefan Saroiu wrote: > > The application still gets 20% of the CPU which is quite good actually. > > Although I'm not familiar with Druschel's work, I'm not sure whether > > better scheduling will help me here. > > > > I've been toying with the idea to change the driver to raise interrupts > > only once every 100 packets or something like that. Currently it is 1 > > interrupt per 1 packet. > > Ouch. No wonder you're having problems. You definitely want to implement > one of coalesced interrupt delivery or polled device access. In theory, > we have both in 4.x and 5.x, but support for coalesced delivery is on a > per-card basis. 5.x will allow you do to the kinds of things you want > (eventually) once the network stack is fine-grained enough, but it sounds > like the big problem is the driver model. I believe fxp and em drivers > support this, and might be a good model to look at. You might want to > consider posting to freebsd-net for pointers in this space. I missed this part. Your best bet is coelesced interrupt delivery in hardware, which should be handled in all gigabit drivers already. I personally provided patches for soft interrupt coelescing for some of the more popular 10/100 drivers. Basically, it takes advantage of Bill Paul's seperation of interrupt processing into rx_eof and tx_eof routines, and then adds a return code to indicate if they've done any work. If they have, it recalls them from the interrupt routine until there is no more work to do. You can put a high watermark on it by counting the number of times that it does this, and bailing, if that's all it's doing. In addition, you might want to look at Luigi Rizzo's patcehs for polling, and John Lemons patches for diminishing the NETISR requirements by processing some operations until completion at interrupt time (a "poor man's LRP"). Both of these are in -current (5.x). The problem with both John and Luigi's patches when used with a user space program is that they do not provide weighted fair share scheduling to ensure that a user space program that requires an arbitrary number of cycles to run gets the CPU time it needs. In effect, you must manually tune the CPU time ratio so that the user space program doesn't starvel, and likewise so that you don't stall the kernel polling of packets "because it's time to do user space processing", if you don't have any user space processing to do. The Druschel and Aron references I made specifically deal with this issue the way the Mogul reference I made suggests that you deal with it, which is to measure the queue depth to user space, and disable interrupts when it hits a high watermark, and reenable them when it hits a low watermark (indicating that the user space application has processed down the data to be processed). Using LRP itself has the additional benefit of removing latency from the processing path. The improved performance is by about a factor of 4 (measured, it went from 7,000 connections a second to 32,000 connections a second, processing through the TCP stack to completion, as opposed to at NETISR, and that's without the SYN cache stuff). If you simply don't have time to do this work, because your work is tangentially related (i.e. you need the performance in order to investigate a network on wich other research is taking place), then FreeBSD may not be the answer. You could apply the LRP patches to FreeBSD rather trivially; here are the patches for FreeBSD 4.3 (these are a port of the Rice University code, made by Duke University): http://www.cs.duke.edu/~anderson/freebsd/rescon/ I dislike resource containers, because they are primarily an accounting niceity which has no bearing on an embedded system application, other than to slow it down. If you want to ignore this problem entirely, then you might want to use QLinux, instead. QLinux uses a number of the ideas I've noted so far in this thread (plus a couple others, which I personally consider to be bad ideas) to improve the overall performance. Here is a pointer to QLinux: http://lass.cs.umass.edu/~lass/software/qlinux/ In general, if you hit performance problems with QLinux, throwing more CPUs at the problem isn't going to help you out any more there than it does in FreeBSD. The problem you have is not ammenable to being forcibly scaled by being spammed with more hardware. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 3 13:11:58 2002 Delivered-To: freebsd-smp@freebsd.org Received: from falcon.prod.itd.earthlink.net (falcon.mail.pas.earthlink.net [207.217.120.74]) by hub.freebsd.org (Postfix) with ESMTP id 7068237B405; Wed, 3 Apr 2002 13:11:48 -0800 (PST) Received: from pool0355.cvx40-bradley.dialup.earthlink.net ([216.244.43.100] helo=mindspring.com) by falcon.prod.itd.earthlink.net with esmtp (Exim 3.33 #1) id 16ss2j-0004Cy-00; Wed, 03 Apr 2002 13:11:45 -0800 Message-ID: <3CAB6FF9.27C6288C@mindspring.com> Date: Wed, 03 Apr 2002 13:11:21 -0800 From: Terry Lambert X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: John Baldwin Cc: Jake Burkholder , smp@FreeBSD.ORG, "Andrew R. Reiter" Subject: Re: Where to initialize certain locks... References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org John Baldwin wrote: > On 03-Apr-2002 Terry Lambert wrote: > > John Baldwin wrote: > >> We need mutexes prior to mi_startup(). > > > > There are really two problems with this theory: > > > > 1) You can't use them before the mutex system is initialized, > > which happens from mi_startup. > > Incorrect. It happens prior to mi_startup(). You missed my followup to myself, where I pointed out that the initi routine could be called anytime, and wasn't married to mi_startup. 8-). > > 2) There are no contended resources until after mi_startup > > starts the other CPU(s). > > But there is lots of code that we don't want to have to do stuff like: > > if (!cold) > mtx_lock(&foo_mtx); > else > /* just wing it */ > > Easier to just always acquire the locks and init them early enough. I understood that at the time I made the posting; I was (figuratively) sticking my tongue out at locking data vs. critical sectioning. 8^p In any case, the foo_init() could be called as early as you wanted; all it does is manipulate data in statically initialized data structures to create the initial linked list that can be extended at runtime, to solve the problem that started this thread. The only limit on it is that you can't call it before the subsystems on which it relies; so if initializing a mutex allocates memory or pages or does other evil, then you have to wait until that code is up (which is true, with or without my proposal). -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Apr 3 19:21:45 2002 Delivered-To: freebsd-smp@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id 52EDB37B41D for ; Wed, 3 Apr 2002 19:21:41 -0800 (PST) Received: from fledge.watson.org (fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.11.6/8.11.6) with SMTP id g343LUw82333; Wed, 3 Apr 2002 22:21:31 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Wed, 3 Apr 2002 22:21:30 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Terry Lambert Cc: Stefan Saroiu , freebsd-smp@FreeBSD.org Subject: Re: Interrupt Handlers and Multiple CPUs In-Reply-To: <3CAB674A.F6BC6F9D@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Wed, 3 Apr 2002, Terry Lambert wrote: > In addition, you might want to look at Luigi Rizzo's patcehs for > polling, and John Lemons patches for diminishing the NETISR requirements > by processing some operations until completion at interrupt time (a > "poor man's LRP"). Both of these are in -current (5.x). I believe polling was MFC'd to -STABLE, so may even be in 4.5-RELEASE. Jonathan's patches for processing to completion in the interrupt thread have not been committed, I believe. His time seems to have evaporated somewhere. Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 4 9:38:39 2002 Delivered-To: freebsd-smp@freebsd.org Received: from earth.hub.org (earth.hub.org [64.49.215.11]) by hub.freebsd.org (Postfix) with ESMTP id A445437B420; Thu, 4 Apr 2002 09:38:13 -0800 (PST) Received: from localhost.localdomain (earth.hub.org [64.49.215.11]) by localhost (Postfix) with ESMTP id 2FCD010350B; Thu, 4 Apr 2002 13:38:07 -0400 (AST) Received: from earth.hub.org (earth.hub.org [64.49.215.11]) by earth.hub.org (Postfix) with ESMTP id CD574103506; Thu, 4 Apr 2002 13:38:06 -0400 (AST) Date: Thu, 4 Apr 2002 13:38:06 -0400 (AST) From: "Marc G. Fournier" To: freebsd-questions@freebsd.org Cc: freebsd-scsi@freebsd.org, Subject: IBM eSeries 440 Server(s) ... Message-ID: <20020404133537.W20291-100000@mail1.hub.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Hi ... I'm looking at picking up one of the above to replace one of our older machines ... plan on starting with just 2xXeon1400CPU and 4Gig of RAM, but moving up to the full 8+32 as budget allows for it ... has anyone played with one of these with FreeBSD? Has there been any work done on the onboard SCSI controller? SMP work well on these? any caveats against them? Thanks ... To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 4 13:51: 2 2002 Delivered-To: freebsd-smp@freebsd.org Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by hub.freebsd.org (Postfix) with ESMTP id 1001237B41A; Thu, 4 Apr 2002 13:50:58 -0800 (PST) Received: (from dillon@localhost) by apollo.backplane.com (8.11.6/8.9.1) id g34LovP40157; Thu, 4 Apr 2002 13:50:57 -0800 (PST) (envelope-from dillon) Date: Thu, 4 Apr 2002 13:50:57 -0800 (PST) From: Matthew Dillon Message-Id: <200204042150.g34LovP40157@apollo.backplane.com> To: Jake Burkholder , John Baldwin , freebsd-smp@FreeBSD.ORG Subject: per-cpu stats counter support comitted. References: <200203292305.g2TN5nX75021@apollo.backplane.com> <20020330001929.H40695@locore.ca> <200203301920.g2UJKxL80198@apollo.backplane.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Ok guys, I've comitted the work. This combined with the commit Bruce just made to the userret()/ast() code removes all memory/cache stalls from the syscall critical path. two-process getuid() tests, SMP build: per-process getuid()/sec ++cnt.v_syscall 853000 PCPU_LAZY_INC(cnt.v_syscall) 1000000 +17.2% PCPU + Bruce's userret() code 1238216 +23.8% Total boost: +45%. This is a significant improvement, folks! In anycase, please feel free to convert other vmeter counters from atomic_add_int() to PCPU_LAZY_INC(). Remember, not all counters can be converted... things like cnt.v_free_min, for example, cannot be converted because the VM system needs to get at the value quickly. Check before you change. We should probably split out the statistics that cannot be made per-cpu. -Matt Matthew Dillon To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 4 15:48:45 2002 Delivered-To: freebsd-smp@freebsd.org Received: from mtiwmhc23.worldnet.att.net (mtiwmhc23.worldnet.att.net [204.127.131.48]) by hub.freebsd.org (Postfix) with ESMTP id 66C2037B41A for ; Thu, 4 Apr 2002 15:48:40 -0800 (PST) Received: from webmail.worldnet.att.net ([204.127.135.75]) by mtiwmhc23.worldnet.att.net (InterMail vM.4.01.03.27 201-229-121-127-20010626) with SMTP id <20020404234839.HZIN8815.mtiwmhc23.worldnet.att.net@webmail.worldnet.att.net> for ; Thu, 4 Apr 2002 23:48:39 +0000 Received: from [206.124.1.129] by webmail.worldnet.att.net; Thu, 04 Apr 2002 23:47:57 +0000 From: webinfo@att.net To: freebsd-smp@freebsd.org Subject: New FreeBSD forums site Date: Thu, 04 Apr 2002 23:47:57 +0000 X-Mailer: AT&T Message Center Version 1 (Mar 27 2002) Message-Id: <20020404234839.HZIN8815.mtiwmhc23.worldnet.att.net@webmail.worldnet.att.net> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Greetings: Would it be possible to add us to your page under "Web Sites of interest" please: http://people.freebsd.org/~fsmp/SMP/SMP.html We are a new FreeBSD forums site and would appreciate your help! We are "FreeBSD Forums" at http://www.freebsdforums.org. We have active SMP discussions under our Hardware forum. Best Wishes, Dan dan@freebsdforums.org ------------------------------------------------ Get the award winning ISP, AT&T WorldNet Service http://download.att.net/webtag To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Apr 4 16: 2:47 2002 Delivered-To: freebsd-smp@freebsd.org Received: from revolt.poohsticks.org (revolt.poohsticks.org [63.227.60.74]) by hub.freebsd.org (Postfix) with ESMTP id 964C437B41C for ; Thu, 4 Apr 2002 16:02:43 -0800 (PST) Received: from revolt.poohsticks.org (localhost [127.0.0.1]) by revolt.poohsticks.org (8.11.3/8.11.3) with ESMTP id g3502VB30130; Thu, 4 Apr 2002 17:02:31 -0700 (MST) (envelope-from drew@revolt.poohsticks.org) Message-Id: <200204050002.g3502VB30130@revolt.poohsticks.org> To: webinfo@att.net Cc: freebsd-smp@FreeBSD.ORG Subject: Re: New FreeBSD forums site In-reply-to: Your message of "Thu, 04 Apr 2002 23:47:57 GMT." <20020404234839.HZIN8815.mtiwmhc23.worldnet.att.net@webmail.worldnet.att.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <30127.1017964951.1@revolt.poohsticks.org> Date: Thu, 04 Apr 2002 17:02:31 -0700 From: Drew Eckhardt Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org In message <20020404234839.HZIN8815.mtiwmhc23.worldnet.att.net@webmail.worldnet .att.net>, webinfo@att.net writes: >We have active SMP discussions under our Hardware >forum. Web forums are an abomination that should be stopped. Unlike mailing lists: - They're not easily archived by individual users to allow searches with reasonable tools - They require users to periodically poll for new messages. If I have to check once a day, I might answer a post or two and forget the rest. With mailing lists, if I'm waiting for a compile to finish I'm likely to respond to the most recent interesting message. - They require users to be online to read messages (this is bad in the backwards places that charge my the minute for dialup internet access) Existing implementations have additional disadvantages compared to mailing lists, including - Bad search facilities - Poor support for quoting - No support for using a reasonable text editor - Slow performance - Bad browser compatability (there are a couple that crash netscape) - Showing all messages in a thread in chronological order, including ones you've already seen -- Home Page For those who do, no explanation is necessary. For those who don't, no explanation is possible. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Apr 5 8:32:23 2002 Delivered-To: freebsd-smp@freebsd.org Received: from cable.net.co (cr2006817333.cable.net.co [200.68.173.33]) by hub.freebsd.org (Postfix) with SMTP id EFAED37B41B for ; Fri, 5 Apr 2002 08:31:50 -0800 (PST) From: Subject: Laser toner cartridges save up to 40% from retail price Date: Vie, 5 Abr 2002 11:08:37 Message-Id: <449.498901.415546@cable.net.co> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" To: undisclosed-recipients:; Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Please forward to the person responsible for purchasing your laser printer supplies. GT toner supplies To order by phone: please call our US toll Free number 1-866-237-7397 order by email: gt@cable.net.co email removal: gt@cable.net.co subject: remove Laser printer toner cartridges, copier and fax cartridges save up 40% from retail price. University and/or school purchase orders welcome. (no credit approval required) Pay by check, c.o.d, or purchase order (net 30 days). ITEM MAKE MODEL PRICE 1 ----- Hp ---- Laserjet series 4l, 4p (74a)----------- $39.00 2 ----- Hp ---- Laserjet series 1100 (92a)------------- $39.00 3 ----- Hp ---- Laserjet series 2 (95a)---------------- $30.00 4 ----- Hp ---- Laserjet series 2p (75a)--------------- $49.00 5 ----- Hp ---- Laserjet series 5p,5mp,6p,6mp (03a) --- $39.00 6 ----- Hp ---- Laserjet series 5si,800 (09a) --------- $89.00 7 ----- Hp ---- Laserjet series 2100 (96a) ------------ $65.00 8 ----- Hp ---- Laserjet series 8100 (82x) ------------ $129.00 9 ----- Hp ---- Laserjet series 5l,6l (3906a) --------- $35.00 10 ---- Hp ---- Laserjet series 4v -------------------- $89.00 11 ---- Hp ---- Laserjet series 4000 (27x) ------------ $65.00 12 ---- Hp ---- Laserjet series 3si,4si (91a) --------- $49.00 13 ---- Hp ---- Laserjet series 4,4m,5,5m (98a) ------- $40.00 14 ---- Hp ---- Laserjet series 5000 (29x) ------------ $89.00 15 ---- Hp ---- Laserfax 500, 700 (fx1) --------------- $44.00 16 ---- Hp ---- Laserfax 5000, 7000 (fx2) ------------- $49.00 17 ---- Hp ---- Laserfax (fx3) ------------------------ $49.00 18 ---- Hp ---- Laserfax (fx4) ------------------------ $49.00 19 ---- Lexmark-Optra 4019,4029 high yield ------------ $79.00 20 ---- Lexmark-Optra r 4039,4049 high yield ---------- $99.00 21 ---- Lexmark-Optra e ------------------------------- $49.00 22 ---- Lexmark-Optra n ------------------------------- $105.00 23 ---- Lexmark-Optra s ------------------------------- $145.00 24 ---- Lexmark-Optra t ------------------------------- $180.00 25 ---- Epson Action laser 7000,7500,8000,9000 ------ $99.00 26 ---- Epson Action laser 1000,1500 --------------- $99.00 27 ---- Panasonic-Nec series 2 models 90 and 95 ------- $99.00 28 ---- Apple - Laser writer pro 600,16/600 ----------- $40.00 29 ---- Apple - Laser writer select 300,320,360 ------- $64.00 30 ---- Apple - Laser writer 300,320 ------------------ $49.00 31 ---- Apple - Laser writer nt,2nt ------------------- $49.00 32 ---- Apple - Laser writer 12/640 ------------------- $69.00 33 ---- Canon - Fax laserclass 4000 (fx3) ------------- $49.00 34 ---- Canon - Fax laserclass 5000,6000,7000 (fx2) --- $44.00 35 ---- Canon - Laserfax 5000,7000 (fx2) -------------- $44.00 36 ---- Canon - Laserfax 8500,9000 (fx4) -------------- $44.00 37 ---- Canon - Copier pc3,6re,7 & 11 (a30) ----------- $59.00 38 ---- Canon - Copier 300,320,700,720 and 760 (e-40) $79.00 30 Day unlimited warranty included on all products. If you are ordering by e-mail or c.o.d. please fill out an order form with the following information: phone number company name first and last name street address city, state zip code If you are ordering by purchase order please fill out an order form with the following information: purchase order number phone number company or school name shipping address and billing address city, state zip code All trade marks and brand names listed above are property of the respective holders and used for descriptive purposes only. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message