From owner-freebsd-smp Sun Jun 24 15:21:25 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mail.utcorp.net (x-montana.utcorp.com [146.145.135.26]) by hub.freebsd.org (Postfix) with ESMTP id 1000C37B401 for ; Sun, 24 Jun 2001 15:21:22 -0700 (PDT) (envelope-from kseel@utcorp.com) Received: from x-kspc.utcorp.com ([146.145.135.17] helo=utcorp.com) by mail.utcorp.net with esmtp (Exim 3.03 #1) id 15EIHy-000Ces-00 for smp@FreeBSD.org; Sun, 24 Jun 2001 18:23:30 -0400 Message-ID: <3B3669C1.ACC9DE02@utcorp.com> Date: Sun, 24 Jun 2001 18:29:22 -0400 From: Kurt Seel X-Mailer: Mozilla 4.76 [en] (X11; U; FreeBSD 4.2-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: smp@FreeBSD.org Subject: re: Athlon AMD SMP Runs References: Content-Type: multipart/alternative; boundary="------------EB81F30463D0C203BA59AFAB" Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org --------------EB81F30463D0C203BA59AFAB Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit John Baldwin wrote: > Completely stock -current as of Thursday on a dual Athlon test box graciously > provided by AMD with the prodding of David O`Brien. > > > dmesg Might this mean that we can use those oh-so-fast dual athlon w/ DDR memory any time soon? As in will this support be back ported to -stable? I always get a giggle when AMD eats Intel's lunch :-) -- In theory, there is no difference between theory and practice. In practice, however, there is. - Albert Einstein --------------EB81F30463D0C203BA59AFAB Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit John Baldwin wrote:
Completely stock -current as of Thursday on a dual Athlon test box graciously
provided by AMD with the prodding of David O`Brien.

> dmesg

 Might this mean that we can use those oh-so-fast dual
athlon w/ DDR memory any time soon? As in will this
support be back ported to -stable?
 I always get a giggle when AMD eats Intel's lunch :-) 
-- 
 In theory, there is no difference between theory
and practice. In practice, however, there is.
 - Albert Einstein
  --------------EB81F30463D0C203BA59AFAB-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Jun 25 13: 4:10 2001 Delivered-To: freebsd-smp@freebsd.org Received: from peorth.iteration.net (peorth.iteration.net [208.190.180.178]) by hub.freebsd.org (Postfix) with ESMTP id 8B0B037B406 for ; Mon, 25 Jun 2001 13:04:06 -0700 (PDT) (envelope-from keichii@iteration.net) Received: by peorth.iteration.net (Postfix, from userid 1001) id 3E4F859229; Mon, 25 Jun 2001 15:04:00 -0500 (CDT) Date: Mon, 25 Jun 2001 15:04:00 -0500 From: "Michael C . Wu" To: Kurt Seel Cc: smp@FreeBSD.org Subject: Re: Athlon AMD SMP Runs Message-ID: <20010625150400.C14321@peorth.iteration.net> Reply-To: "Michael C . Wu" References: <3B3669C1.ACC9DE02@utcorp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3B3669C1.ACC9DE02@utcorp.com>; from kseel@utcorp.com on Sun, Jun 24, 2001 at 06:29:22PM -0400 X-PGP-Fingerprint: 5025 F691 F943 8128 48A8 5025 77CE 29C5 8FA1 2E20 X-PGP-Key-ID: 0x8FA12E20 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Sun, Jun 24, 2001 at 06:29:22PM -0400, Kurt Seel scribbled: | John Baldwin wrote: | > Completely stock -current as of Thursday on a dual Athlon test box graciously | > provided by AMD with the prodding of David O`Brien. | | Might this mean that we can use those oh-so-fast dual | athlon w/ DDR memory any time soon? As in will this | support be back ported to -stable? | I always get a giggle when AMD eats Intel's lunch :-) It already works. Please read previous messages. -- +-----------------------------------------------------------+ | keichii@iteration.net | keichii@freebsd.org | | http://iteration.net/~keichii | Yes, BSD is a conspiracy. | +-----------------------------------------------------------+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jun 28 14: 1:59 2001 Delivered-To: freebsd-smp@freebsd.org Received: from peorth.iteration.net (peorth.iteration.net [208.190.180.178]) by hub.freebsd.org (Postfix) with ESMTP id 55C4B37B405 for ; Thu, 28 Jun 2001 14:01:55 -0700 (PDT) (envelope-from keichii@iteration.net) Received: by peorth.iteration.net (Postfix, from userid 1001) id E9AFB59229; Thu, 28 Jun 2001 16:01:53 -0500 (CDT) Date: Thu, 28 Jun 2001 16:01:53 -0500 From: "Michael C . Wu" To: "Daniel C. Sobral" Cc: Kurt Seel , smp@FreeBSD.ORG Subject: Re: Athlon AMD SMP Runs Message-ID: <20010628160153.A55976@peorth.iteration.net> Reply-To: "Michael C . Wu" References: <3B3669C1.ACC9DE02@utcorp.com> <20010625150400.C14321@peorth.iteration.net> <3B3B988F.3060108@tcoip.com.br> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3B3B988F.3060108@tcoip.com.br>; from daniel.sobral@tcoip.com.br on Thu, Jun 28, 2001 at 05:50:23PM -0300 X-PGP-Fingerprint: 5025 F691 F943 8128 48A8 5025 77CE 29C5 8FA1 2E20 X-PGP-Key-ID: 0x8FA12E20 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Thu, Jun 28, 2001 at 05:50:23PM -0300, Daniel C. Sobral scribbled: | Michael C . Wu wrote: | > On Sun, Jun 24, 2001 at 06:29:22PM -0400, Kurt Seel scribbled: | > | John Baldwin wrote: | > | > Completely stock -current as of Thursday on a dual Athlon test box graciously | > | > provided by AMD with the prodding of David O`Brien. | > | | > | Might this mean that we can use those oh-so-fast dual | > | athlon w/ DDR memory any time soon? As in will this | > | support be back ported to -stable? | > | I always get a giggle when AMD eats Intel's lunch :-) | > | > It already works. Please read previous messages. | ... | > | support be back ported to -stable? | | It seems clear to me that the question was not answered by the previous | messages. It already works on both RELENG_4(4.3-STABLE) and HEAD (5.0-CURRENT). -- +-----------------------------------------------------------+ | keichii@iteration.net | keichii@freebsd.org | | http://iteration.net/~keichii | Yes, BSD is a conspiracy. | +-----------------------------------------------------------+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jun 28 19:15:15 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id DF73237B409; Thu, 28 Jun 2001 19:15:10 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5T2F5J02161; Fri, 29 Jun 2001 02:15:05 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Fri, 29 Jun 2001 02:15:04 +0000 (GMT) From: "E.B. Dreger" To: freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: libc_r locking... why? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Please pardon the cross-posting; I'd rather keep responses on whichever list is more appropriate. Why are bind(2), accept(2), kevent(2), etc. wrapped in libc_r? I thought that the spl() calls prevented kernel recursion in the current SMP system, and that a mutex handled reentrance in SMPng. [Please correct me if/where I am mistaken.] I can understand things like malloc(3), lseek(2), read(2), and write(2) being serialized, but I'm confused about [some of the other] syscall wrappers. Can somebody please elaborate, or direct me to a reference? Big TIA, Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jun 28 19:30: 2 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mta5.rcsntx.swbell.net (mta5.rcsntx.swbell.net [151.164.30.29]) by hub.freebsd.org (Postfix) with ESMTP id BD82A37B403; Thu, 28 Jun 2001 19:29:56 -0700 (PDT) (envelope-from chris@holly.calldei.com) Received: from holly.calldei.com ([208.191.149.190]) by mta5.rcsntx.swbell.net (Sun Internet Mail Server sims.3.5.2000.03.23.18.03.p10) with ESMTP id <0GFO00KIQ6SY0A@mta5.rcsntx.swbell.net>; Thu, 28 Jun 2001 21:26:59 -0500 (CDT) Received: (from chris@localhost) by holly.calldei.com (8.11.1/8.9.3) id f5T2TgL63943; Thu, 28 Jun 2001 21:29:43 -0500 (CDT envelope-from chris) Date: Thu, 28 Jun 2001 21:28:56 -0500 From: Chris Costello Subject: Re: libc_r locking... why? In-reply-to: ; from eddy+public+spam@noc.everquick.net on Fri, Jun 29, 2001 at 02:15:04AM +0000 To: "E.B. Dreger" Cc: freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Reply-To: chris@calldei.com Message-id: <20010628212856.E55395@holly.calldei.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-disposition: inline User-Agent: Mutt/1.2.5i References: Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Friday, June 29, 2001, E.B. Dreger wrote: > Please pardon the cross-posting; I'd rather keep responses on whichever > list is more appropriate. > > Why are bind(2), accept(2), kevent(2), etc. wrapped in libc_r? Currently, the pthreads implementation is done entirely in userland. This means that a syscall which would normally block needs to have code in it to check if it would block (write(2) is a really simple example of this), and if it would, schedule another thread (cancelling, or blocking, the calling thread) to run, and eventually get back to the thread which is blocking on write, check for/read more data, cancel again, etc., until the requested amount of data has been read or an error occurs. This example, of course, applies to instances where write() is called on a file descriptor which does _NOT_ have O_NONBLOCK set. kevent(2), bind(2), accept(2) etc. all do the same thing for the same reasons. -- +-------------------+-----------------------------------------+ | Chris Costello | Advanced design: | | chris@calldei.com | Upper management doesn't understand it. | +-------------------+-----------------------------------------+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jun 28 21:21: 5 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id 5ED0E37B409; Thu, 28 Jun 2001 21:20:55 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5T4Kqw03424; Fri, 29 Jun 2001 04:20:52 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Fri, 29 Jun 2001 04:20:51 +0000 (GMT) From: "E.B. Dreger" To: Chris Costello Cc: freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? In-Reply-To: <20010628212856.E55395@holly.calldei.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > Date: Thu, 28 Jun 2001 21:28:56 -0500 > From: Chris Costello > > > Please pardon the cross-posting; I'd rather keep responses on whichever > > list is more appropriate. > > Currently, the pthreads implementation is done entirely in > userland. This means that a syscall which would normally block > needs to have code in it to check if it would block (write(2) > is a really simple example of this), and if it would, schedule > another thread (cancelling, or blocking, the calling thread) to > run, and eventually get back to the thread which is blocking on > write, check for/read more data, cancel again, etc., until the > requested amount of data has been read or an error occurs. So it's a thunk/kludge not only to enforce "proper" behavior, but also to prevent the process from blocking and stalling other threads? This makes sense. > This example, of course, applies to instances where write() is > called on a file descriptor which does _NOT_ have O_NONBLOCK set. > kevent(2), bind(2), accept(2) etc. all do the same thing for the > same reasons. The reason that I asked is because I'm writing a program that uses rfork() in the same manner as the new rfork_thread(). I couldn't understand the need to wrap kevent(2), bind(2), or accept(2)... In my mind, I was thinking "data integrity", trying to prevent processes in the same "thread family" from stepping on one another. Blocking is not a problem; where I can't use non-blocking calls, I use a worker thread. I guess that I was looking at man pages and bits of libc_r code without understanding the pthread implementation. I knew that it was userland, but I thought that it created multiple processes... if this is not the case, then I was apparently comparing apples to mangoes. Am I correct that libc_r does _not_ use multiple processes to create threads? Grepping for "fork" in *.c files under /usr/src/lib/libc_r leads me to believe that this is so... ...I think that, with your prompting, I've answered my question. Thanks, Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Jun 28 21:34:59 2001 Delivered-To: freebsd-smp@freebsd.org Received: from mta5.rcsntx.swbell.net (mta5.rcsntx.swbell.net [151.164.30.29]) by hub.freebsd.org (Postfix) with ESMTP id CEB7837B40C; Thu, 28 Jun 2001 21:34:54 -0700 (PDT) (envelope-from chris@holly.calldei.com) Received: from holly.calldei.com ([208.191.149.190]) by mta5.rcsntx.swbell.net (Sun Internet Mail Server sims.3.5.2000.03.23.18.03.p10) with ESMTP id <0GFO00255CL7R0@mta5.rcsntx.swbell.net>; Thu, 28 Jun 2001 23:31:56 -0500 (CDT) Received: (from chris@localhost) by holly.calldei.com (8.11.1/8.9.3) id f5T4Ych64342; Thu, 28 Jun 2001 23:34:39 -0500 (CDT envelope-from chris) Date: Thu, 28 Jun 2001 23:33:56 -0500 From: Chris Costello Subject: Re: libc_r locking... why? In-reply-to: ; from eddy+public+spam@noc.everquick.net on Fri, Jun 29, 2001 at 04:20:51AM +0000 To: "E.B. Dreger" Cc: freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Reply-To: chris@calldei.com Message-id: <20010628233355.F55395@holly.calldei.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-disposition: inline User-Agent: Mutt/1.2.5i References: <20010628212856.E55395@holly.calldei.com> Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Friday, June 29, 2001, E.B. Dreger wrote: > Am I correct that libc_r does _not_ use multiple processes to create > threads? Grepping for "fork" in *.c files under /usr/src/lib/libc_r leads > me to believe that this is so... That's correct. It's implemented using setjmp/longjmp, and storing stack pointers and the like in thread-specific data structures. -- +-------------------+--------------------------------+ | Chris Costello | A bug in the code is worth two | | chris@calldei.com | in the documentation. | +-------------------+--------------------------------+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 0:50:15 2001 Delivered-To: freebsd-smp@freebsd.org Received: from avocet.mail.pas.earthlink.net (avocet.mail.pas.earthlink.net [207.217.121.50]) by hub.freebsd.org (Postfix) with ESMTP id C860037B406; Fri, 29 Jun 2001 00:50:03 -0700 (PDT) (envelope-from tlambert2@mindspring.com) Received: from mindspring.com (dialup-209.245.136.253.Dial1.SanJose1.Level3.net [209.245.136.253]) by avocet.mail.pas.earthlink.net (EL-8_9_3_3/8.9.3) with ESMTP id AAA20681; Fri, 29 Jun 2001 00:49:58 -0700 (PDT) Message-ID: <3B3C3346.E5496485@mindspring.com> Date: Fri, 29 Jun 2001 00:50:30 -0700 From: Terry Lambert Reply-To: tlambert2@mindspring.com X-Mailer: Mozilla 4.7 [en]C-CCK-MCD {Sony} (Win98; U) X-Accept-Language: en MIME-Version: 1.0 To: "E.B. Dreger" Cc: Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org "E.B. Dreger" wrote: [ ... wrapped fd using functions in libc_r ... ] > So it's a thunk/kludge not only to enforce "proper" > behavior, but also to prevent the process from blocking > and stalling other threads? This makes sense. It also permits locks on the descriptors, to ensure that one thread does not modify a descriptor out from under another thread while it is "blocked" on some outstanding operation. [ ... ] > The reason that I asked is because I'm writing a program > that uses rfork() in the same manner as the new > rfork_thread(). I couldn't understand the need to wrap > kevent(2), bind(2), or accept(2)... > > In my mind, I was thinking "data integrity", trying to > prevent processes in the same "thread family" from stepping > on one another. Blocking is not a problem; where I can't > use non-blocking calls, I use a worker thread. The threads scheduler is in user space. It converts a blobking call into a non-blocking call plus a context switch. THus blocking _IS_ a problem. > I guess that I was looking at man pages and bits of > libc_r code without understanding the pthread implementation. > I knew that it was userland, but I thought that it created > multiple processes... if this is not the case, then I was > apparently comparing apples to mangoes. This is not the case. The user space threads library does what the original idea of threads was intended to do, before people started treating it as the only hammer they had to pound on the SMP problem with in order to achieve SMP scalability: it utilizes the full quantum of the process, and minimizes context switch overhead. Kernel threads don't do either of these things well, in almost all existing implementations out there. > Am I correct that libc_r does _not_ use multiple processes > to create threads? Yes. All threads run in a single process. The threads are not intended as a workaround for the SMP scalability problem. Note that you are not going to be able to combine your rfork approach with this, if your resulting processes end up running on different CPUs: this is because the locking primitives in the libc_r library do _NOT_ use the "lock" prefix on the "cmpxchg" instruction, which means that multiple processors are not forced to a rendevous, there's no IPI, and the TLB and L1 cache shootdown isn't moderated by the cache MP 1.4 specification cache coherency protocol, and thus the locks it uses are _NOT_ MP safe. If you "need" kernel threads, look at the Linux kernel threads in the ports collection (it's a kernel module that builds and installs as a package). You probably don't, since performance of kernel threads is really only about a 20% increment, if you implement them the SVR4 or Solaris (pre-2.7) or Linux way. It's probably better to implement with FreeBSD threads as they currently exist, and get massive SMP scalability when KSE's are brought into the source tree. -- Terry To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 8:20:10 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id 24F6F37B401; Fri, 29 Jun 2001 08:19:55 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5TFJmY08950; Fri, 29 Jun 2001 15:19:48 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Fri, 29 Jun 2001 15:19:47 +0000 (GMT) From: "E.B. Dreger" To: Terry Lambert Cc: Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? In-Reply-To: <3B3C3346.E5496485@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org (Warning: rather long message) > Date: Fri, 29 Jun 2001 00:50:30 -0700 > From: Terry Lambert > > [ ... wrapped fd using functions in libc_r ... ] [ fd locking, to prevent chopping feet from beneath ] As-needed serialization to prevent breakage = "proper" behavior. I should have been more clear. > The threads scheduler is in user space. It converts a > blobking call into a non-blocking call plus a context > switch. THus blocking _IS_ a problem. Bad wording on my part again; perhaps "a problem that I [think that] I have handled" is better. I'm use nb calls if possible; else I have a long-running worker thread. After my recent question regarding AIO, it looks like it's time to bite the bullet and use that as well. > [ ... thinking that pthreads used multiple processes ... ] > > This is not the case. So I've learned. I'm glad that I didn't use pthreads, then. :-) > The user space threads library does what the original > idea of threads was intended to do, before people started > treating it as the only hammer they had to pound on the > SMP problem with in order to achieve SMP scalability: it > utilizes the full quantum of the process, and minimizes > context switch overhead. Kernel threads don't do either > of these things well, in almost all existing implementations > out there. Agreed on all counts. I'm tempted to continue eschewing the pthread library. I've unrolled code, and store state info in a purpose-specific FSM control block. Maybe I reinvented the wheel, but it wasn't that difficult. > > Am I correct that libc_r does _not_ use multiple processes > > to create threads? > > Yes. All threads run in a single process. The threads > are not intended as a workaround for the SMP scalability > problem. A good thing, IMHO. I was starting to look at libc_r to check my work; I _prefer_ launching multiple processess for SMP scalability, and having an untainted threading model. > Note that you are not going to be able to combine your > rfork approach with this, if your resulting processes > end up running on different CPUs: this is because the Running processes on multiple CPUs is one goal. [ libc_r locks don't assert "lock", not MP-safe ] So the "lock" prefix is the only way to enforce cache coherency? Do you have handy a good reference on IPIs, other than the kernel APIC code (and, of course, Google and NorthernLight searches)? Good to know, but, I'm not using libc_r... I was looking at existing code to help me double-check mine as I go. I'm synchronizing processes with a "giant lock" token that each process cooperatively passes to the next... to simplify: who_has_lock++ ; who_has_lock %= process_count ; Each processes' critical path first checks to see if it holds the token; if so, it performs the tasks that require it, such as locking a finer-grained lock or mutex. It then passes the token, and continues through its critical path. If a thread has nothing to do, I nanosleep(2) to prevent the critical path from degenerating to an extended spin. I'm considering using socketpair(2)s, with a process blocking indefinitely on read(2) until another process write(2)s to awaken it... > If you "need" kernel threads, look at the Linux kernel > threads in the ports collection (it's a kernel module > that builds and installs as a package). You probably > don't, since performance of kernel threads is really only Correct me if I'm wrong, but the only place in my model that really might benefit from kthreads would be the scheduling? i.e., rather than screwing around with nanosleep(2) or socket calls, I could cut the cruft and interact more directly with the scheduler via kthread mechanisms? > about a 20% increment, if you implement them the SVR4 or > Solaris (pre-2.7) or Linux way. It's probably better to > implement with FreeBSD threads as they currently exist, > and get massive SMP scalability when KSE's are brought > into the source tree. KSEs... where can I read up? Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 8:29:49 2001 Delivered-To: freebsd-smp@freebsd.org Received: from ringworld.nanolink.com (ringworld.nanolink.com [195.24.48.13]) by hub.freebsd.org (Postfix) with SMTP id CE62637B403 for ; Fri, 29 Jun 2001 08:29:27 -0700 (PDT) (envelope-from roam@orbitel.bg) Received: (qmail 3303 invoked by uid 1000); 29 Jun 2001 15:33:52 -0000 Date: Fri, 29 Jun 2001 18:33:52 +0300 From: Peter Pentchev To: "E.B. Dreger" Cc: Terry Lambert , Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? Message-ID: <20010629183352.D535@ringworld.oblivion.bg> Mail-Followup-To: "E.B. Dreger" , Terry Lambert , Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG References: <3B3C3346.E5496485@mindspring.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from eddy+public+spam@noc.everquick.net on Fri, Jun 29, 2001 at 03:19:47PM +0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Fri, Jun 29, 2001 at 03:19:47PM +0000, E.B. Dreger wrote: > > The threads scheduler is in user space. It converts a > > blobking call into a non-blocking call plus a context > > switch. THus blocking _IS_ a problem. > > Bad wording on my part again; perhaps "a problem that I [think > that] I have handled" is better. I'm use nb calls if possible; > else I have a long-running worker thread. I hope you understand that when the worker thread blocks, it's the process that blocks, and none of the other threads can run until the end of the syscall. > After my recent question regarding AIO, it looks like it's time > to bite the bullet and use that as well. G'luck, Peter -- If this sentence were in Chinese, it would say something else. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 8:44:16 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id D593537B403; Fri, 29 Jun 2001 08:44:08 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5TFi4D09415; Fri, 29 Jun 2001 15:44:04 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Fri, 29 Jun 2001 15:44:03 +0000 (GMT) From: "E.B. Dreger" To: Peter Pentchev Cc: Terry Lambert , Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? In-Reply-To: <20010629183352.D535@ringworld.oblivion.bg> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > Date: Fri, 29 Jun 2001 18:33:52 +0300 > From: Peter Pentchev > > > > The threads scheduler is in user space. It converts a > > > blobking call into a non-blocking call plus a context > > > switch. THus blocking _IS_ a problem. > > > > Bad wording on my part again; perhaps "a problem that I [think > > that] I have handled" is better. I'm use nb calls if possible; > > else I have a long-running worker thread. > > I hope you understand that when the worker thread blocks, > it's the process that blocks, and none of the other threads ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Yes. > it's the process that blocks, and none of the other threads ^^^^^^^^^^^^^^^^^^^^^^^^^ > can run until the end of the syscall. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Again, I am *not* using pthreads. Worker thread = totally separate process, created via rfork(2). One process blocks, others continue running. To reiterate: I'm *not* using pthreads or libc_r. I wanted to check my work, and began looking at libc_r code, under the faulty ass-umption that it ran multiple processes. Now that I know that threads are implemented in a single process, and that blocking calls are thunked to non-blocking calls, the locking that I originally questioned makes sense. Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 10:36:33 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id B045437B403; Fri, 29 Jun 2001 10:36:26 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5THaQB11205; Fri, 29 Jun 2001 17:36:26 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Fri, 29 Jun 2001 17:36:25 +0000 (GMT) From: "E.B. Dreger" To: freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: CPU affinity hinting Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org (Cross-posting again... I'm willing to be larted with a herring if this is unacceptable for the content presented.) I was thinking about CPU affinity on SMP systems.... the following is on-list brainstorming. Take a two-way box running 10 httpd and 10 smtpd processes. Assuming equal CPU time requirements, it would make sense to bind httpd to one CPU, and smtpd to the other. Simple, but not realistic. Maybe smtpd requires more CPU time. Fine... limit one processor to smtpd, run leftover smtpd on the other CPU, and run httpd _only_ on the processor handling leftover smtpd. Or consider ten instances of a single program that uses four processes, sort of like squid * 10: It would make more sense to have similar processes grouped on the same CPU. After watching processes switch CPUs via "top", I got to thinking... could there be, and would it be useful to have, a mechanism where processes could tell the kernel "my magic number is 6819732", and the kernel would try to keep all processes with said magic number on the same CPU? Is this "solution" worse than the problem (cache thrash and switching CPUs)? I suppose that the kernel could do a quick, numerically-simple hash of the ELF metadata, as opposed to program-specified magic. This would handle the httpd/smtpd case, with less fear of magic number collisions, but not rfork(2)ed threads. Or, instead of hashing ELF metadata, the kernel could compute a hash based on all IP ports bind(2)ed by the program within the first few seconds of operation. (Obviously unsuitable for short-lived programs, but those could probably be handled via least-busy CPU assignment.) Perhaps a hybrid approach: cpu_hint = hash(elf_metadata, hint) % num_cpus ; where "hint" is specified by the process in a group? Thoughts? Criticisms? Flames? Beatings with a stinky fish? Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 11:33:22 2001 Delivered-To: freebsd-smp@freebsd.org Received: from srv1.cosmo-project.de (srv1.cosmo-project.de [213.83.6.106]) by hub.freebsd.org (Postfix) with ESMTP id 41CEF37B403; Fri, 29 Jun 2001 11:33:16 -0700 (PDT) (envelope-from ticso@mail.cicely.de) Received: from mail.cicely.de (cicely20 [10.1.1.22]) by srv1.cosmo-project.de (8.11.0/8.11.0) with ESMTP id f5TIXD658164; Fri, 29 Jun 2001 20:33:13 +0200 (CEST) Received: (from ticso@localhost) by mail.cicely.de (8.11.0/8.11.0) id f5TIXvJ17293; Fri, 29 Jun 2001 20:33:57 +0200 (CEST) Date: Fri, 29 Jun 2001 20:33:51 +0200 From: Bernd Walter To: "E.B. Dreger" Cc: Peter Pentchev , Terry Lambert , Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? Message-ID: <20010629203351.A16557@cicely20.cicely.de> References: <20010629183352.D535@ringworld.oblivion.bg> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from eddy+public+spam@noc.everquick.net on Fri, Jun 29, 2001 at 03:44:03PM +0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Fri, Jun 29, 2001 at 03:44:03PM +0000, E.B. Dreger wrote: > Again, I am *not* using pthreads. Worker thread = totally separate > process, created via rfork(2). One process blocks, others continue > running. I can't see how you make shure that on SMP systems all CPUs have the same meaning from memory content. Normaly you would use a mutex or similar before accessing a data range from another thread which also enshures that the CPU specific caches and buffers are syncronised. If you don't do this it may happen that you write a variable and another thread uses this variable using another CPU before the first CPU has writen this memory seeable for others and works with an outdated content. A fresh rforked process with the same virtual memory should at least see the version at the time of the rfork, so there is no problem if you don't modify the common used content after rfork. -- B.Walter COSMO-Project http://www.cosmo-project.de ticso@cicely.de Usergroup info@cosmo-project.de To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 11:44:50 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id 3C2BF37B401; Fri, 29 Jun 2001 11:44:45 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5TIiXI12135; Fri, 29 Jun 2001 18:44:33 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Fri, 29 Jun 2001 18:44:29 +0000 (GMT) From: "E.B. Dreger" To: Bernd Walter Cc: Peter Pentchev , Terry Lambert , Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? In-Reply-To: <20010629203351.A16557@cicely20.cicely.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > Date: Fri, 29 Jun 2001 20:33:51 +0200 > From: Bernd Walter > > I can't see how you make shure that on SMP systems all CPUs have > the same meaning from memory content. > Normaly you would use a mutex or similar before accessing a data range > from another thread which also enshures that the CPU specific caches > and buffers are syncronised. > If you don't do this it may happen that you write a variable and > another thread uses this variable using another CPU before the first > CPU has writen this memory seeable for others and works with an > outdated content. Passing a token between threads. When a thread has the token, it may assert a lock or a mutex on an object. Again, I subscribe to threads being lightweight; cooperative sharing is better than preemptive or trying to grab a lock before another thread does. Any good references on MP standard? Is the lock prefix the only way to force cache coherency? Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 12:17:54 2001 Delivered-To: freebsd-smp@freebsd.org Received: from srv1.cosmo-project.de (srv1.cosmo-project.de [213.83.6.106]) by hub.freebsd.org (Postfix) with ESMTP id 5B9AC37B405; Fri, 29 Jun 2001 12:17:44 -0700 (PDT) (envelope-from ticso@mail.cicely.de) Received: from mail.cicely.de (cicely20 [10.1.1.22]) by srv1.cosmo-project.de (8.11.0/8.11.0) with ESMTP id f5TJHe658356; Fri, 29 Jun 2001 21:17:42 +0200 (CEST) Received: (from ticso@localhost) by mail.cicely.de (8.11.0/8.11.0) id f5TJIO217420; Fri, 29 Jun 2001 21:18:24 +0200 (CEST) Date: Fri, 29 Jun 2001 21:18:18 +0200 From: Bernd Walter To: "E.B. Dreger" Cc: Bernd Walter , Peter Pentchev , Terry Lambert , Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? Message-ID: <20010629211818.A17309@cicely20.cicely.de> References: <20010629203351.A16557@cicely20.cicely.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from eddy+public+spam@noc.everquick.net on Fri, Jun 29, 2001 at 06:44:29PM +0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Fri, Jun 29, 2001 at 06:44:29PM +0000, E.B. Dreger wrote: > > Date: Fri, 29 Jun 2001 20:33:51 +0200 > > From: Bernd Walter > > > > I can't see how you make shure that on SMP systems all CPUs have > > the same meaning from memory content. > > Normaly you would use a mutex or similar before accessing a data range > > from another thread which also enshures that the CPU specific caches > > and buffers are syncronised. > > If you don't do this it may happen that you write a variable and > > another thread uses this variable using another CPU before the first > > CPU has writen this memory seeable for others and works with an > > outdated content. > > Passing a token between threads. When a thread has the token, it may > assert a lock or a mutex on an object. Again, I subscribe to threads > being lightweight; cooperative sharing is better than preemptive or trying > to grab a lock before another thread does. A Token may not be enough because writes may be reordered. Usually a mutex is a lock with some kind of memory barrier. If you can fetch the lock on a CPU you know that the CPU previous owning the lock has flushed everything up to the point of coherence of what was written until the lock was released. Memory barriers and the read-modify-write operations (or locked operations like on ALPHA) are accessible from user code - but I don't know of any platform independend functions. > Any good references on MP standard? Is the lock prefix the only way to > force cache coherency? A good explanation for this kind of thing was in "Programming with POSIX Threads" in Chapter 3.4 "Memory visible between threads". I know you are not usung pthreads but the memory problems are the same. > > > Eddy > > --------------------------------------------------------------------------- > > Brotsman & Dreger, Inc. > EverQuick Internet Division > > Phone: +1 (316) 794-8922 Wichita/(Inter)national > Phone: +1 (785) 865-5885 Lawrence > > --------------------------------------------------------------------------- > > Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) > From: A Trap > To: blacklist@brics.com > Subject: Please ignore this portion of my mail signature. > > These last few lines are a trap for address-harvesting spambots. Do NOT > send mail to , or you are likely to be blocked. > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-hackers" in the body of the message -- B.Walter COSMO-Project http://www.cosmo-project.de ticso@cicely.de Usergroup info@cosmo-project.de To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 12:56:56 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id C049A37B401; Fri, 29 Jun 2001 12:56:47 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5TJue813575; Fri, 29 Jun 2001 19:56:40 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Fri, 29 Jun 2001 19:56:40 +0000 (GMT) From: "E.B. Dreger" To: Bernd Walter Cc: freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? In-Reply-To: <20010629211818.A17309@cicely20.cicely.de> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org (Personal CCs trimmed; back to Bernd and cross-posting -smp and -hackers) > Date: Fri, 29 Jun 2001 21:18:18 +0200 > From: Bernd Walter > > Passing a token between threads. When a thread has the token, it may > > assert a lock or a mutex on an object. Again, I subscribe to threads > > being lightweight; cooperative sharing is better than preemptive or > > trying to grab a lock before another thread does. > > A Token may not be enough because writes may be reordered. > Usually a mutex is a lock with some kind of memory barrier. But it _is_ locked. The thread_with_token++ ; thread_with_token %= num_threads ; was oversimplified. It's more like xorl %ecx,%ecx movl thread_with_token,%eax incl %eax cmpl %eax,num_threads movzl %ecx,%eax lock movl %eax,thread_with_token to pass the token, where thread_with_token is in shared memory. Each process does the above. When a process has the token, it's safe to claim mutexes and such without worry of another thread (without token) accessing simultaneously. Mutex/lock ops also have lock asserted. If this is inadequate, then I need to do some head-scratching. > If you can fetch the lock on a CPU you know that the CPU previous > owning the lock has flushed everything up to the point of coherence > of what was written until the lock was released. Here is where I want to learn more about cache coherency, inter-processor interrupts, and APIC programming. I'd imagine that the latter two are lower-level than I'd be using, but I still want to know the "how and why" beneath the scenes. > Memory barriers and the read-modify-write operations (or locked > operations like on ALPHA) are accessible from user code - but I don't > know of any platform independend functions. Nor do I. > > Any good references on MP standard? Is the lock prefix the only way to > > force cache coherency? > > A good explanation for this kind of thing was in "Programming with POSIX > Threads" in Chapter 3.4 "Memory visible between threads". I'll have to check it out. I [believe that I] understand about the inherent races in SMP, but more reading is always better... However, I'm still interested in x86-specific coherency mechanisms. Any info? > I know you are not usung pthreads but the memory problems are the same. Correct. I just wanted to make certain that we were on the same page, no pun intended. Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 14:14:17 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id 36AFD37B405; Fri, 29 Jun 2001 14:14:09 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5TLE7U14825; Fri, 29 Jun 2001 21:14:07 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Fri, 29 Jun 2001 21:14:06 +0000 (GMT) From: "E.B. Dreger" To: Matthew Rogers Cc: freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: RE: CPU affinity hinting In-Reply-To: <002001c100d8$2c406c40$a2962640@bear> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > Date: Fri, 29 Jun 2001 13:14:58 -0700 > From: Matthew Rogers > > Why not just use First in line, Next processor available ? Then you > wouldn't care what processor did which task. That was my question: Would the added complexity of "CPU affinity hinting" be worth the reduction in cache misses and switching processes, by preventing long-running processes from constantly switching CPUs? FILNPA is fine for short-lived processes, but longer-running ones switch CPUs, perhaps unnecessarily. > Hmm, maybe even have each processor a dedicated memory space, and > programmable functionality. > > Oops, that's a Field Programmable Gate Array, and there going to make > Legacy computing look stupid. FPGAs, mmmm. Transputers, mmmm. Neuromatrix, mmmm. > In my mind, you have a need for multiprocessing Non-specific and > Specific tasking. > > In some ways we are multiprocessing anyway on some level. Videocard 3d > processing, sound card. You mean that Winmodems and main memory-based video aren't the keys to high performance? You mean that Intel is being silly when they justify faster chips by saying "now you can eliminate three $20 DSPs by buying our latest architecture"? :-) > So why do we need a GOD chip, ie the "chipset" controlling access to > processors and busses ? > > That's because that's the way it was done before the 286. > > Time to leave the bus. :) Arguably so from a hardware standpoint. But, in the mean time, I was trying to think of ways to help SMP performance. :-) Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 18:39:30 2001 Delivered-To: freebsd-smp@freebsd.org Received: from RedDust.bluesky.net.au (CPE-61-9-143-86.vic.bigpond.net.au [61.9.143.86]) by hub.freebsd.org (Postfix) with ESMTP id 8639F37B401; Fri, 29 Jun 2001 18:39:24 -0700 (PDT) (envelope-from receiver@blueskybbs.yi.org) Received: from localhost (receiver@localhost) by RedDust.bluesky.net.au (8.11.4/8.11.0) with ESMTP id f5U1XJP08832; Sat, 30 Jun 2001 11:33:20 +1000 (EST) (envelope-from receiver@blueskybbs.yi.org) Date: Sat, 30 Jun 2001 11:33:18 +1000 (EST) From: Idea Receiver To: Terry Lambert Cc: "E.B. Dreger" , Chris Costello , freebsd-smp@FreeBSD.ORG, freebsd-hackers@FreeBSD.ORG Subject: Re: libc_r locking... why? In-Reply-To: <3B3C3346.E5496485@mindspring.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Fri, 29 Jun 2001, Terry Lambert wrote: > "E.B. Dreger" wrote: > If you "need" kernel threads, look at the Linux kernel > threads in the ports collection (it's a kernel module > that builds and installs as a package). You probably > don't, since performance of kernel threads is really only > about a 20% increment, if you implement them the SVR4 or > Solaris (pre-2.7) or Linux way. It's probably better to > implement with FreeBSD threads as they currently exist, > and get massive SMP scalability when KSE's are brought > into the source tree. > just a quick question... I konw KSE will brought in after SMPng. but it will be really helpful to konw when it will first appear in the source tree? or what other OS can help with SMP vs pthread problem? thx. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 19:44:49 2001 Delivered-To: freebsd-smp@freebsd.org Received: from peorth.iteration.net (peorth.iteration.net [208.190.180.178]) by hub.freebsd.org (Postfix) with ESMTP id D1FB137B405; Fri, 29 Jun 2001 19:44:44 -0700 (PDT) (envelope-from keichii@iteration.net) Received: by peorth.iteration.net (Postfix, from userid 1001) id 6E9C759229; Fri, 29 Jun 2001 21:44:43 -0500 (CDT) Date: Fri, 29 Jun 2001 21:44:43 -0500 From: "Michael C . Wu" To: "E.B. Dreger" Cc: Matthew Rogers , freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: CPU affinity hinting Message-ID: <20010629214443.A69846@peorth.iteration.net> Reply-To: "Michael C . Wu" Mail-Followup-To: "Michael C . Wu" , "E.B. Dreger" , Matthew Rogers , freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org References: <002001c100d8$2c406c40$a2962640@bear> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from eddy+public+spam@noc.everquick.net on Fri, Jun 29, 2001 at 09:14:06PM +0000 X-PGP-Fingerprint: 5025 F691 F943 8128 48A8 5025 77CE 29C5 8FA1 2E20 X-PGP-Key-ID: 0x8FA12E20 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Fri, Jun 29, 2001 at 09:14:06PM +0000, E.B. Dreger scribbled: | > Date: Fri, 29 Jun 2001 13:14:58 -0700 | > From: Matthew Rogers The issue is a lot more complicated than what you think. This actually is a big issue in our future SMP implementation. There are two types of processor affinity: user-configurable and system automated. We have no implementation of the former, and alfred-vm has a semblance of the latter. Please wait patiently..... Michael -- +-----------------------------------------------------------+ | keichii@iteration.net | keichii@freebsd.org | | http://iteration.net/~keichii | Yes, BSD is a conspiracy. | +-----------------------------------------------------------+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 21:35:36 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id 5969B37B403; Fri, 29 Jun 2001 21:35:27 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5U4ZOG18906; Sat, 30 Jun 2001 04:35:24 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Sat, 30 Jun 2001 04:35:23 +0000 (GMT) From: "E.B. Dreger" To: "Michael C . Wu" Cc: Matthew Rogers , freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: CPU affinity hinting In-Reply-To: <20010629214443.A69846@peorth.iteration.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > Date: Fri, 29 Jun 2001 21:44:43 -0500 > From: Michael C . Wu > > The issue is a lot more complicated than what you think. How so? I know that idleproc and the new ipending / threaded INTs enter the picture... and, after seeing the "HLT benchmark" page, it would appear that simply doing nothing is sometimes better than doing something, although I'm still scratching my head over that... > This actually is a big issue in our future SMP implementation. I presumed as much; the examples I gave were trivial. I also assume that memory allocation is a major issue... to not waste time with inter-CPU locking, I'd assume that memory would be split into pools, a la Hoard. Maybe start with approx. NPROC count equally-sized pools, which are roughly earmarked per hypothetical process. For example: If MAXUSERS=80 --> NPROC=1300, a machine with 256 MB might use 192 kB initial granularity for mmap() requests, giving 128 MB to each processor as a first approximation. Now, no locking is needed on mmap() until a given CPU's "pool" hits low water, and steals from another pool. This would hopefully be infrequent, particularly assuming that memory allocations would be distributed roughly equally between CPUs. I'm assuming that memory allocations are 1:1 mappable wrt processes. Yes, I know that's faulty and oversimplified, particularly for things like buffers and filesystem cache. > There are two types of processor affinity: user-configurable > and system automated. We have no implementation of the former, Again, why not "hash(sys_auto, user_config) % NCPU"? Identical processes would be on same CPU unless perturbed by user_config. Collisions from identical user_config values in unrelated processes would be less likely because of the sys_auto pertubation. Granted: It Is Always More Complicated. (TM) But for a first pass... > and alfred-vm has a semblance of the latter. Please wait > patiently..... Or, if impatient, would one continue to brainstorm, not expect a response (i.e., not get disappointed when something basic is posted), and track -current after the destabilization? :-) Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 22:47:55 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id F325C37B405; Fri, 29 Jun 2001 22:47:50 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f5U5lnV19618; Sat, 30 Jun 2001 05:47:49 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Sat, 30 Jun 2001 05:47:49 +0000 (GMT) From: "E.B. Dreger" To: freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: Quick question: AIO / SMP / process-based threading Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org Quick question(s): 1. Is AIO SMP-safe? 2. If not, how could one force coherency? (Read and rewrite locked a word from each cache line?) Is it worth the effort, or should one not use AIO across process boundaries? I'm asking primarily about 4.x, unless anyone has good guesses of how 5.x will be. ;-) I'll also keep an eye out for KSEs... thanks to Terry and others for alerting me to those. (KSEs really answer most of my recent questions, but I don't think that I can wait that long, nor do I have the kernel background to really offer any assistance in the KSE project...) TIA, Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Jun 29 22:57:55 2001 Delivered-To: freebsd-smp@freebsd.org Received: from peorth.iteration.net (peorth.iteration.net [208.190.180.178]) by hub.freebsd.org (Postfix) with ESMTP id 752F437B401; Fri, 29 Jun 2001 22:57:50 -0700 (PDT) (envelope-from keichii@iteration.net) Received: by peorth.iteration.net (Postfix, from userid 1001) id A3CB959229; Sat, 30 Jun 2001 00:57:49 -0500 (CDT) Date: Sat, 30 Jun 2001 00:57:49 -0500 From: "Michael C . Wu" To: "E.B. Dreger" Cc: freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Quick question: AIO / SMP / process-based threading Message-ID: <20010630005749.A72545@peorth.iteration.net> Reply-To: "Michael C . Wu" Mail-Followup-To: "Michael C . Wu" , "E.B. Dreger" , freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from eddy+public+spam@noc.everquick.net on Sat, Jun 30, 2001 at 05:47:49AM +0000 X-PGP-Fingerprint: 5025 F691 F943 8128 48A8 5025 77CE 29C5 8FA1 2E20 X-PGP-Key-ID: 0x8FA12E20 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Sat, Jun 30, 2001 at 05:47:49AM +0000, E.B. Dreger scribbled: | 1. Is AIO SMP-safe? AIO is not safe, SMP or not. | 2. If not, how could one force coherency? (Read and rewrite locked | a word from each cache line?) Is it worth the effort, or should | one not use AIO across process boundaries? Don't use it. | I'm asking primarily about 4.x, unless anyone has good guesses of | how 5.x will be. ;-) | | I'll also keep an eye out for KSEs... thanks to Terry and others | for alerting me to those. (KSEs really answer most of my recent | questions, but I don't think that I can wait that long, nor do I | have the kernel background to really offer any assistance in the | KSE project...) | KSE won't be available for a long time. -- +-----------------------------------------------------------+ | keichii@iteration.net | keichii@freebsd.org | | http://iteration.net/~keichii | Yes, BSD is a conspiracy. | +-----------------------------------------------------------+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Jun 30 20:28:43 2001 Delivered-To: freebsd-smp@freebsd.org Received: from sneakerz.org (sneakerz.org [216.33.66.254]) by hub.freebsd.org (Postfix) with ESMTP id B8FE637B401; Sat, 30 Jun 2001 20:28:39 -0700 (PDT) (envelope-from bright@sneakerz.org) Received: by sneakerz.org (Postfix, from userid 1092) id 30E755D020; Sat, 30 Jun 2001 22:28:29 -0500 (CDT) Date: Sat, 30 Jun 2001 22:28:29 -0500 From: Alfred Perlstein To: "Michael C . Wu" Cc: "E.B. Dreger" , freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Quick question: AIO / SMP / process-based threading Message-ID: <20010630222829.E84523@sneakerz.org> References: <20010630005749.A72545@peorth.iteration.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2i In-Reply-To: <20010630005749.A72545@peorth.iteration.net>; from keichii@iteration.net on Sat, Jun 30, 2001 at 12:57:49AM -0500 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org * Michael C . Wu [010630 14:05] wrote: > On Sat, Jun 30, 2001 at 05:47:49AM +0000, E.B. Dreger scribbled: > | 1. Is AIO SMP-safe? > > AIO is not safe, SMP or not. > > | 2. If not, how could one force coherency? (Read and rewrite locked > | a word from each cache line?) Is it worth the effort, or should > | one not use AIO across process boundaries? > > Don't use it. Can you point to some specific PRs about this or crashdumps before (or at least while) taking pot shots at the AIO implementation? thanks, -- -Alfred Perlstein [alfred@freebsd.org] Ok, who wrote this damn function called '??'? And why do my programs keep crashing in it? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Jun 30 20:51:26 2001 Delivered-To: freebsd-smp@freebsd.org Received: from a.mx.everquick.net (a.mx.everquick.net [216.89.137.3]) by hub.freebsd.org (Postfix) with ESMTP id A052637B405; Sat, 30 Jun 2001 20:51:21 -0700 (PDT) (envelope-from eddy+public+spam@noc.everquick.net) Received: from localhost (eddy@localhost) by a.mx.everquick.net (8.10.2/8.10.2) with ESMTP id f613pB429235; Sun, 1 Jul 2001 03:51:11 GMT X-EverQuick-No-Abuse: Report any e-mail abuse to Date: Sun, 1 Jul 2001 03:51:10 +0000 (GMT) From: "E.B. Dreger" To: Alfred Perlstein Cc: "Michael C . Wu" , freebsd-smp@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: Quick question: AIO / SMP / process-based threading In-Reply-To: <20010630222829.E84523@sneakerz.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org > Date: Sat, 30 Jun 2001 22:28:29 -0500 > From: Alfred Perlstein > > Can you point to some specific PRs about this or crashdumps before > (or at least while) taking pot shots at the AIO implementation? In the mean time, until somebody can substantiate that claim... is AIO SMP safe? I see that aiocb.aio_buf is declared as "volatile", so I would presume so. I just want to be sure that, if an aio call runs on one CPU, another CPU can access *aio_buf and be 100% certain that the data are coherent. aio_buf = mmap() using MAP_HASSEMAPHORE -- good idea, bad idea, pointless? TIA, Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: +1 (316) 794-8922 Wichita/(Inter)national Phone: +1 (785) 865-5885 Lawrence --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to , or you are likely to be blocked. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message