From owner-freebsd-arch Sun Nov 7 2:13:43 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 20EDB14A2E for ; Sun, 7 Nov 1999 02:13:40 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id LAA23477 for ; Sun, 7 Nov 1999 11:13:38 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id LAA16010 for freebsd-arch@freebsd.org; Sun, 7 Nov 1999 11:13:37 +0100 (MET) Received: from alpo.whistle.com (alpo.whistle.com [207.76.204.38]) by hub.freebsd.org (Postfix) with ESMTP id 1D9F314C0D for ; Sun, 7 Nov 1999 02:12:00 -0800 (PST) (envelope-from julian@whistle.com) Received: from current1.whiste.com (current1.whistle.com [207.76.205.22]) by alpo.whistle.com (8.9.1a/8.9.1) with ESMTP id CAA57945; Sun, 7 Nov 1999 02:11:56 -0800 (PST) Date: Sun, 7 Nov 1999 02:11:56 -0800 (PST) From: Julian Elischer To: "Daniel C. Sobral" Cc: Chris Csanady , Terry Lambert , freebsd-arch@freebsd.org Subject: Re: Threads goals version III In-Reply-To: <38252EE8.5F627E51@newsguy.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG The trouble is that 'quanti' are only useful in discussion when related to the rest of the processes and the system load. On Sun, 7 Nov 1999, Daniel C. Sobral wrote: > Chris Csanady wrote: > > > > I don't see the problem. So, you ask for two quantum--they will end > > up being on the same CPU in this case. They may still be scheduled in > > different classes as you please. > > > > The idea is that you don't want to do this in general, because you > > have more context switches than necessary, and increased process > > migration. Asking for the same number of quanta as CPU's is simply > > the optimal case. > > > > Or, at least that is how I understand it.. > > /me mmmms > > Let's see what Terry says. What you describe makes sense, but he > won't be able to use so few words. :-) > > -- > Daniel C. Sobral (8-DCS) > dcs@newsguy.com > dcs@freebsd.org > > What y'all wanna do? > Wanna be hackers? Code crackers? Slackers > Wastin' time with all the chatroom yakkers? > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Nov 7 2:47:20 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 8BACC14CEA for ; Sun, 7 Nov 1999 02:47:18 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id LAA23701 for ; Sun, 7 Nov 1999 11:47:17 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id LAA16098 for freebsd-arch@freebsd.org; Sun, 7 Nov 1999 11:47:16 +0100 (MET) Received: from alpo.whistle.com (alpo.whistle.com [207.76.204.38]) by hub.freebsd.org (Postfix) with ESMTP id 3313B14CEA for ; Sun, 7 Nov 1999 02:47:11 -0800 (PST) (envelope-from julian@whistle.com) Received: from current1.whiste.com (current1.whistle.com [207.76.205.22]) by alpo.whistle.com (8.9.1a/8.9.1) with ESMTP id CAA58415; Sun, 7 Nov 1999 02:45:06 -0800 (PST) Date: Sun, 7 Nov 1999 02:45:04 -0800 (PST) From: Julian Elischer To: "Russell L. Carter" Cc: Poul-Henning Kamp , freebsd-arch@freebsd.org Subject: Re: Threads models and FreeBSD. (Next Step) In-Reply-To: <19991102143819.5D0F23B@chomsky.pinyon.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Tue, 2 Nov 1999, Russell L. Carter wrote: > %In message <381EDBCE.FD7FBA68@vigrid.com>, "Daniel M. Eischen" writes: > % > %>> >Disagree. I want lightweight processes to have their own quantum > %>> >not limited (in total) to the parent process quantum. > %>> > %>> That would clearly kill the "lightweight" in "lightweight process"... > %> > %>That doesn't mean they each have to have the same quantum as a non-MT > %>process. > % > %That has nothing to do with it. > % > %There is not much point in making a lightweight process facility > %if the resulting processes are not lightweight. > > Then the application developer should choose libc_r threads. > > There isn't much point to doing this effort if the > pthread_*sched* functions don't actually mean much in the > global context. > > People building large scale distributed objects that > are also high performance require fine grained schedulability > of individual threads. I can provide references that demonstrate > how low level thread scheduling architecture > affect high level services. > > Solaris, HP, MVS, and Linux support this to varying degrees now. > The RT-OSs more or less do, though some fail in surprising ways. > Linux is surprisingly good. > > Put another way, cramming a process's threads into a single > scheduling quanta significantly diminishes the suitability of FreeBSD > as a platform for building high performance and/or RT CORBA apps. Let's rephrase this.. Firstly 'A single scheduling quanta' is probably a misleading statement. firstly it would be 'A group of quanta' as the process may be at a priority where it gets a lot of time and there may be only one other process with only 1 quanta allocated to it.. 99/100 is pretty close to all you can get.. secondly, I see a need to be able to effectively 'fork off a separate family of threads' that are treated as a separate scheduling entity. Thsi only makes sense at all if the threads in that group are explicitly bound to that group, and other threads cannot arbitrarily wander in and out of the group. The sum of the processing quanta for the original and the new class would be the same as a user process forking to run 2 processes in parallel, and would be limited by the same mechanisms. The second process could not increaes its priority higher than the parent, just like nice(2) at the moment. (You wuld need root priority to do that) > > Regards, > Russell > > > > %-- > %Poul-Henning Kamp FreeBSD coreteam member > %phk@FreeBSD.ORG "Real hackers run -current on their laptop." > %FreeBSD -- It will take a long time before progress goes too far! > % > % > % > % > %To Unsubscribe: send mail to majordomo@FreeBSD.org > %with "unsubscribe freebsd-arch" in the body of the message > % > > > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Nov 7 6:29:57 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id CD8A314BE1 for ; Sun, 7 Nov 1999 06:29:45 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id PAA25470 for ; Sun, 7 Nov 1999 15:29:43 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id PAA16970 for freebsd-arch@freebsd.org; Sun, 7 Nov 1999 15:29:43 +0100 (MET) Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 70F8114BE1 for ; Sun, 7 Nov 1999 06:29:26 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from vigrid.com (pm3-pt36.pcnet.net [206.105.29.110]) by pcnet1.pcnet.com (8.8.7/PCNet) with ESMTP id JAA14359; Sun, 7 Nov 1999 09:24:41 -0500 (EST) Message-ID: <38258BFA.B7EA3ECB@vigrid.com> Date: Sun, 07 Nov 1999 09:26:02 -0500 From: "Daniel M. Eischen" X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 4.0-CURRENT i386) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer Cc: "Russell L. Carter" , Poul-Henning Kamp , freebsd-arch@freebsd.org Subject: Re: Threads models and FreeBSD. (Next Step) References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Julian Elischer wrote: > > On Tue, 2 Nov 1999, Russell L. Carter wrote: [...] > > Put another way, cramming a process's threads into a single > > scheduling quanta significantly diminishes the suitability of FreeBSD > > as a platform for building high performance and/or RT CORBA apps. > > Let's rephrase this.. > > Firstly 'A single scheduling quanta' is probably a misleading statement. > > firstly it would be > 'A group of quanta' as the process may be at a priority where it gets a > lot of time and there may be only one other process with only 1 quanta > allocated to it.. > 99/100 is pretty close to all you can get.. > > secondly, I see a need to be able to effectively 'fork off a separate > family of threads' that are treated as a separate scheduling entity. > Thsi only makes sense at all if the threads in that group are explicitly > bound to that group, and other threads cannot arbitrarily wander in and > out of the group. Except for very narrow windows where threads holding critical resources (internal to the threads library) are continued until they release the resource. I can also see the need for threads to cross scheduling group boundaries in order to support priority protection and priority ceiling mutexes. > > The sum of the processing quanta for the original and the new class would > be the same as a user process forking to run 2 processes in parallel, and > would be limited by the same mechanisms. The second process could not > increaes its priority higher than the parent, just like nice(2) at the > moment. (You wuld need root priority to do that) Yes! As long as we can "obtain a new quantum" with the ability (as root) to raise priority or enter a different scheduling class, then I am satisified :-) Dan Eischen eischen@vigrid.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Nov 7 6:42:39 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 7D56E14C2D for ; Sun, 7 Nov 1999 06:42:29 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id PAA25573 for ; Sun, 7 Nov 1999 15:42:28 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id PAA17033 for freebsd-arch@freebsd.org; Sun, 7 Nov 1999 15:42:28 +0100 (MET) Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 95AA314C2D for ; Sun, 7 Nov 1999 06:42:13 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from vigrid.com (pm3-pt36.pcnet.net [206.105.29.110]) by pcnet1.pcnet.com (8.8.7/PCNet) with ESMTP id JAA15734; Sun, 7 Nov 1999 09:37:34 -0500 (EST) Message-ID: <38258EFF.399F9D5F@vigrid.com> Date: Sun, 07 Nov 1999 09:38:55 -0500 From: "Daniel M. Eischen" X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 4.0-CURRENT i386) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer , "Russell L. Carter" , freebsd-arch@freebsd.org Subject: Re: Threads models and FreeBSD. (Next Step) References: <38258BFA.B7EA3ECB@vigrid.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel M. Eischen" wrote: > > Except for very narrow windows where threads holding critical resources > (internal to the threads library) are continued until they release the > resource. I can also see the need for threads to cross scheduling group > boundaries in order to support priority protection and priority ceiling ^^^^^^^^^^ ^^^^^^^ Eek! These are one and the same. I meant priority _inheritence_ and priority protection mutexes. Dan Eischen eischen@vigrid.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Nov 7 10: 7:13 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 8AEA914EC7 for ; Sun, 7 Nov 1999 10:07:06 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id TAA27551 for ; Sun, 7 Nov 1999 19:07:05 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id TAA17663 for freebsd-arch@freebsd.org; Sun, 7 Nov 1999 19:07:05 +0100 (MET) Received: from green.myip.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 6D18514FDC; Sun, 7 Nov 1999 10:05:49 -0800 (PST) (envelope-from green@FreeBSD.org) Received: from localhost ([127.0.0.1] ident=green) by green.myip.org with esmtp (Exim 3.02 #1) id 11kEm5-0000RH-00; Sat, 06 Nov 1999 17:57:34 -0500 Date: Sat, 6 Nov 1999 17:57:33 -0500 (EST) From: Brian Fundakowski Feldman X-Sender: green@green.myip.org To: obrien@freebsd.org Cc: arch@freebsd.org Subject: Re: new MD5 option In-Reply-To: <19991106133558.A5539@dragon.nuxi.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Sat, 6 Nov 1999, David O'Brien wrote: > I am adding an option to md5 to reverse the order of output. > > From: > MD5 (/COPYRIGHT) = 7df8bc77dcee71382ea73eb0ec6a9243 > > to: > MD5 7df8bc77dcee71382ea73eb0ec6a9243 /COPYRIGHT > > The reson for this addition is to aid visual diffs. They are much easier > if the MD5 hashes are nicely lined up. I don't understand the issue. If you're going to compare MD5 (/dev/null) = d41d8cd98f00b204e9800998ecf8427e and MD5 (/dev/null) = 93b885adfe0da089cdf634904fd59f71 They are already lined up. If instead you want to compare something like MD5 (foo) = d3b07384d113edec49eaa6238ad5ff00 MD5 (quux) = d3b07a382ec010c01889250fce66fb13 to MD5 (foo) = 1bfe5d8702009e431bad3862b79f5d95 MD5 (quux) = 8687c409c3cc95464d49a7c4dff71e73 They're less lined up, but they look a lot better to a human than MD5 1bfe5d8702009e431bad3862b79f5d95 foo MD5 8687c409c3cc95464d49a7c4dff71e73 quux If for some reason we _really_ need this, what's wrong with a shell transformation instead: transform_md5 () { local line while read line; do printf "MD5\t%s\t%s\n" $(echo "$line" | cut -d' ' -f4) \ done } {"/home/green"}$ md5 /dev/null | transform_md5 MD5 d41d8cd98f00b204e9800998ecf8427e /dev/null What good is all the convenience we have available to reduce bloat if we never use it for anything? > > My question is what flag to use. "-r" for reversed output? "-v" for > visual diff? > > -- > -- David (obrien@NUXI.com) > > -- Brian Fundakowski Feldman \ FreeBSD: The Power to Serve! / green@FreeBSD.org `------------------------------' To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Nov 7 19:55: 9 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 8596514D5C for ; Sun, 7 Nov 1999 19:55:06 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id EAA12227 for ; Mon, 8 Nov 1999 04:55:06 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id EAA19742 for freebsd-arch@freebsd.org; Mon, 8 Nov 1999 04:55:05 +0100 (MET) Received: from smtp01.primenet.com (smtp01.primenet.com [206.165.6.131]) by hub.freebsd.org (Postfix) with ESMTP id 5957014F45 for ; Sun, 7 Nov 1999 19:54:57 -0800 (PST) (envelope-from rcarter@chomsky.Pinyon.ORG) Received: (from daemon@localhost) by smtp01.primenet.com (8.9.3/8.9.3) id UAA03341; Sun, 7 Nov 1999 20:54:49 -0700 (MST) Received: from ip-83-131.prc.primenet.com(207.218.83.131), claiming to be "chomsky.pinyon.org" via SMTP by smtp01.primenet.com, id smtpdAAAmIaOAg; Sun Nov 7 20:54:47 1999 Received: from chomsky.Pinyon.ORG (localhost [127.0.0.1]) by chomsky.pinyon.org (Postfix) with ESMTP id 2161E80; Sun, 7 Nov 1999 20:54:25 -0700 (MST) X-Mailer: exmh version 2.1.0 09/18/1999 To: Julian Elischer Cc: freebsd-arch@freebsd.org, rcarter@pinyon.org Subject: Re: Threads models and FreeBSD. (Next Step) In-Reply-To: Message from Julian Elischer of "Sun, 07 Nov 1999 02:45:04 PST." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Sun, 07 Nov 1999 20:54:25 -0700 From: "Russell L. Carter" Message-Id: <19991108035425.2161E80@chomsky.pinyon.org> Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG % %On Tue, 2 Nov 1999, Russell L. Carter wrote: % [...] %> %> Put another way, cramming a process's threads into a single %> scheduling quanta significantly diminishes the suitability of FreeBSD %> as a platform for building high performance and/or RT CORBA apps. % %Let's rephrase this.. ok, but first, apply: s/single scheduling quanta/single scheduling class/ to the above. Weak minded mistake, sorry. % %Firstly 'A single scheduling quanta' is probably a misleading statement. % %firstly it would be %'A group of quanta' as the process may be at a priority where it gets a %lot of time and there may be only one other process with only 1 quanta %allocated to it.. %99/100 is pretty close to all you can get.. Not hard realtime. This is the "best" that can be hoped for, and is certainly generous. But perhaps it could be tunable, down, what I'm concerned about is over allocation of resources required to meet weak QoS "guarantees" for things like streaming multimedia. And some other stuff. %secondly, I see a need to be able to effectively 'fork off a separate %family of threads' that are treated as a separate scheduling entity. %Thsi only makes sense at all if the threads in that group are explicitly %bound to that group, and other threads cannot arbitrarily wander in and %out of the group. Right. The spec explicitly supports this now. But as a specialization, I can think of no (money-making :-) application that requires a thread's scheduling class to be changed during the thread's lifetime (after initial thread initialization). So I think this restriction is useful. %The sum of the processing quanta for the original and the new class would %be the same as a user process forking to run 2 processes in parallel, and %would be limited by the same mechanisms. The second process could not %increaes its priority higher than the parent, just like nice(2) at the %moment. (You wuld need root priority to do that) Frankly, any unpriviledged user space process should abide by these rules, always. That's not the issue I thought; the issue is whether or not some processes (with root priviledges) in the system could take advantage of the soft realtime capabilities specified by the optional POSIX thread interfaces, and accessible via Peter D.'s modifications to the rtprio interface. Thread libraries such as LinuxThreads can do this, though flawed, today. And on top of those are built, well, look back at my other mails. Daniel alludes to priority inheritence for thread mutexes in his response. There's a lot underneath that issue! Schedulability is at the heart of the problem. It's not at all irrelevant to an OS with a reputation for high performance and reliable content delivery. Regards, Russell To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sun Nov 7 22:31:52 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 6592A14E4D for ; Sun, 7 Nov 1999 22:31:49 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id HAA13521 for ; Mon, 8 Nov 1999 07:31:45 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id HAA20031 for freebsd-arch@freebsd.org; Mon, 8 Nov 1999 07:31:43 +0100 (MET) Received: from relay.nuxi.com (nuxi.cs.ucdavis.edu [169.237.7.38]) by hub.freebsd.org (Postfix) with ESMTP id 7D13114E4D; Sun, 7 Nov 1999 22:31:38 -0800 (PST) (envelope-from obrien@NUXI.com) Received: from dragon.nuxi.com (root@d60-025.leach.ucdavis.edu [169.237.60.25]) by relay.nuxi.com (8.9.3/8.9.3) with ESMTP id WAA38856; Sun, 7 Nov 1999 22:31:37 -0800 (PST) (envelope-from obrien@dragon.nuxi.com) Received: (from obrien@localhost) by dragon.nuxi.com (8.9.3/8.9.1) id WAA62460; Sun, 7 Nov 1999 22:31:37 -0800 (PST) (envelope-from obrien) Date: Sun, 7 Nov 1999 22:31:37 -0800 From: "David O'Brien" To: Brian Fundakowski Feldman Cc: arch@freebsd.org Subject: Re: new MD5 option Message-ID: <19991107223137.C29186@dragon.nuxi.com> Reply-To: obrien@NUXI.com References: <19991106133558.A5539@dragon.nuxi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0i In-Reply-To: ; from green@FreeBSD.org on Sat, Nov 06, 1999 at 05:57:33PM -0500 X-Operating-System: FreeBSD 4.0-CURRENT Organization: The NUXI BSD group X-PGP-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Keyid: 34F9F9D5 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > > I am adding an option to md5 to reverse the order of output. ... > MD5 (foo) = 1bfe5d8702009e431bad3862b79f5d95 > MD5 (quux) = 8687c409c3cc95464d49a7c4dff71e73 > > They're less lined up, but they look a lot better to a human than > > MD5 1bfe5d8702009e431bad3862b79f5d95 foo > MD5 8687c409c3cc95464d49a7c4dff71e73 quux Your opinion. I often MD5 compare two files who's pathname lengths are *significantly* different from each other. While my "-r" output isn't the most pretty, it is certainly easier for my eyes to run down the two hashes to see if they match, than when one starts in column 20 and the other in column 70. Think comparing installed files with the live file system CDROM. > If for some reason we _really_ need this, what's wrong with a shell > transformation instead: Because I don't want to have to install such a shell script on every machine. Unless of course I commit the shell script. > What good is all the convenience we have available to reduce bloat > if we never use it for anything? When I use the flag 80% of the time I use the program, it starts to become required in my eyes. -- -- David (obrien@NUXI.com) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Nov 8 0:15: 1 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 41E9814C17 for ; Mon, 8 Nov 1999 00:14:58 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id JAA14835 for ; Mon, 8 Nov 1999 09:14:57 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id JAA20401 for freebsd-arch@freebsd.org; Mon, 8 Nov 1999 09:14:56 +0100 (MET) Received: from ren.detir.qld.gov.au (ns.detir.qld.gov.au [203.46.81.66]) by hub.freebsd.org (Postfix) with ESMTP id 1E92D14F1D; Mon, 8 Nov 1999 00:14:47 -0800 (PST) (envelope-from syssgm@detir.qld.gov.au) Received: by ren.detir.qld.gov.au; id SAA03125; Mon, 8 Nov 1999 18:14:35 +1000 (EST) Received: from ogre.detir.qld.gov.au(167.123.8.3) by ren.detir.qld.gov.au via smap (3.2) id xma003118; Mon, 8 Nov 99 18:14:07 +1000 Received: from atlas.detir.qld.gov.au (atlas.detir.qld.gov.au [167.123.8.9]) by ogre.detir.qld.gov.au (8.8.8/8.8.7) with ESMTP id SAA24499; Mon, 8 Nov 1999 18:13:18 +1000 (EST) Received: from nymph.detir.qld.gov.au (nymph.detir.qld.gov.au [167.123.10.10]) by atlas.detir.qld.gov.au (8.8.5/8.8.5) with ESMTP id SAA19736; Mon, 8 Nov 1999 18:13:19 +1000 (EST) Received: from nymph.detir.qld.gov.au (localhost.detir.qld.gov.au [127.0.0.1]) by nymph.detir.qld.gov.au (8.8.8/8.8.7) with ESMTP id SAA25572; Mon, 8 Nov 1999 18:13:16 +1000 (EST) (envelope-from syssgm@nymph.detir.qld.gov.au) Message-Id: <199911080813.SAA25572@nymph.detir.qld.gov.au> To: "David O'Brien" Cc: arch@freebsd.org, Brian Fundakowski Feldman , syssgm@detir.qld.gov.au Subject: Re: new MD5 option References: <19991107223137.C29186@dragon.nuxi.com> In-Reply-To: <19991107223137.C29186@dragon.nuxi.com> from "David O'Brien" at "Sun, 07 Nov 1999 22:31:37 -0800" Date: Mon, 08 Nov 1999 18:13:16 +1000 From: Stephen McKay Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Sunday, 7th November 1999, "David O'Brien" wrote: >> > I am adding an option to md5 to reverse the order of output. >... >> MD5 (foo) = 1bfe5d8702009e431bad3862b79f5d95 >> MD5 (quux) = 8687c409c3cc95464d49a7c4dff71e73 >> >> They're less lined up, but they look a lot better to a human than >> >> MD5 1bfe5d8702009e431bad3862b79f5d95 foo >> MD5 8687c409c3cc95464d49a7c4dff71e73 quux BSDI use: 1bfe5d8702009e431bad3862b79f5d95 foo 8687c409c3cc95464d49a7c4dff71e73 quux That is, `hash space name'. No tabs. I think that is neater, cleaner, and all round better. I've always disliked the extraneous junk that comes out of our md5. Perhaps we can just throw the old one away? BTW, they also have a -q option to remove the file name from the output too, leaving a bare md5 hash. Useful. >I often MD5 compare two files who's pathname lengths are >*significantly* different from each other. While my "-r" output isn't >the most pretty, it is certainly easier for my eyes to run down the two >hashes to see if they match, than when one starts in column 20 and the >other in column 70. Think comparing installed files with the live file >system CDROM. I do this sort of comparison all the time too. The current md5 is inadequate. >When I use the flag 80% of the time I use the program, it starts to >become required in my eyes. I hope there is some way to make the new behaviour the default. Stephen. PS Are we discussing this on -arch simply because -current is too noisy? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Nov 8 1:20:44 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 8C527151CF for ; Mon, 8 Nov 1999 01:20:39 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id KAA16749 for ; Mon, 8 Nov 1999 10:20:38 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id KAA20633 for freebsd-arch@freebsd.org; Mon, 8 Nov 1999 10:20:35 +0100 (MET) Received: from m0.cs.berkeley.edu (m0.CS.Berkeley.EDU [128.32.45.176]) by hub.freebsd.org (Postfix) with ESMTP id 9275414F10; Mon, 8 Nov 1999 01:20:27 -0800 (PST) (envelope-from asami@stampede.cs.berkeley.edu) Received: from silvia.hip.berkeley.edu (sji-ca7-01.ix.netcom.com [209.109.235.1]) by m0.cs.berkeley.edu (8.9.3/8.9.3) with ESMTP id BAA50699; Mon, 8 Nov 1999 01:20:25 -0800 (PST) (envelope-from asami@stampede.cs.berkeley.edu) Received: (from asami@localhost) by silvia.hip.berkeley.edu (8.9.3/8.6.9) id BAA83057; Mon, 8 Nov 1999 01:20:17 -0800 (PST) To: Stephen McKay Cc: "David O'Brien" , arch@freebsd.org, Brian Fundakowski Feldman Subject: Re: new MD5 option References: <19991107223137.C29186@dragon.nuxi.com> <199911080813.SAA25572@nymph.detir.qld.gov.au> From: asami@freebsd.org (Satoshi - Ports Wraith - Asami) Date: 08 Nov 1999 01:20:16 -0800 In-Reply-To: Stephen McKay's message of "Mon, 08 Nov 1999 18:13:16 +1000" Message-ID: Lines: 11 X-Mailer: Gnus v5.7/Emacs 20.4 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG * From: Stephen McKay I agree that the new output format is useful, but: * I hope there is some way to make the new behaviour the default. Please. There are gobs of scripts out there written to work with the old format. Changing the default output order will cause mass destruction. ;) Satoshi To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Nov 8 18:39:10 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id F1F1414F8D for ; Mon, 8 Nov 1999 18:39:04 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id DAA07575 for ; Tue, 9 Nov 1999 03:39:03 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id DAA25381 for freebsd-arch@freebsd.org; Tue, 9 Nov 1999 03:39:02 +0100 (MET) Received: from dorifer.heim3.tu-clausthal.de (dorifer.heim3.tu-clausthal.de [139.174.243.252]) by hub.freebsd.org (Postfix) with ESMTP id C152514F8D for ; Mon, 8 Nov 1999 18:38:26 -0800 (PST) (envelope-from olli@dorifer.heim3.tu-clausthal.de) Received: (from olli@localhost) by dorifer.heim3.tu-clausthal.de (8.8.8/8.8.8) id DAA18296 for freebsd-arch@FreeBSD.ORG; Tue, 9 Nov 1999 03:38:25 +0100 (CET) (envelope-from olli) Date: Tue, 9 Nov 1999 03:38:25 +0100 (CET) From: Oliver Fromme Message-Id: <199911090238.DAA18296@dorifer.heim3.tu-clausthal.de> To: freebsd-arch@freebsd.org Subject: Re: new MD5 option Organization: Administration Heim 3 Reply-To: freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Newsreader: TIN [version 1.2 RZTUC(3) PL2] Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG What's wrong with md5 * | awk '{print $4,$2}' ? It doesn't look nice, but it serves the purpose (you can make it look nicer with a bit more effort, too). After all, this is UNIX, and there are already tools to do almost everything. You just have to combine them. Best regards Oliver Fromme -- Oliver Fromme, Leibnizstr. 18/61, 38678 Clausthal, Germany (Info: finger userinfo:olli@dorifer.heim3.tu-clausthal.de) "In jedem Stück Kohle wartet ein Diamant auf seine Geburt" (Terry Pratchett) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Mon Nov 8 23:29:21 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 8060214F29 for ; Mon, 8 Nov 1999 23:29:18 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id IAA09931 for ; Tue, 9 Nov 1999 08:29:16 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id IAA26128 for freebsd-arch@freebsd.org; Tue, 9 Nov 1999 08:29:16 +0100 (MET) Received: from relay.nuxi.com (nuxi.cs.ucdavis.edu [169.237.7.38]) by hub.freebsd.org (Postfix) with ESMTP id A2A3714CF9 for ; Mon, 8 Nov 1999 23:19:49 -0800 (PST) (envelope-from obrien@NUXI.com) Received: from dragon.nuxi.com (root@d60-025.leach.ucdavis.edu [169.237.60.25]) by relay.nuxi.com (8.9.3/8.9.3) with ESMTP id XAA49076; Mon, 8 Nov 1999 23:19:49 -0800 (PST) (envelope-from obrien@dragon.nuxi.com) Received: (from obrien@localhost) by dragon.nuxi.com (8.9.3/8.9.1) id XAA06586; Mon, 8 Nov 1999 23:19:48 -0800 (PST) (envelope-from obrien) Date: Mon, 8 Nov 1999 23:19:48 -0800 From: "David O'Brien" To: Stephen McKay Cc: arch@freebsd.org Subject: Re: new MD5 option Message-ID: <19991108231948.B6537@dragon.nuxi.com> Reply-To: obrien@freebsd.org References: <19991107223137.C29186@dragon.nuxi.com> <199911080813.SAA25572@nymph.detir.qld.gov.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0i In-Reply-To: <199911080813.SAA25572@nymph.detir.qld.gov.au>; from syssgm@detir.qld.gov.au on Mon, Nov 08, 1999 at 06:13:16PM +1000 X-Operating-System: FreeBSD 4.0-CURRENT Organization: The NUXI BSD group X-PGP-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Keyid: 34F9F9D5 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Mon, Nov 08, 1999 at 06:13:16PM +1000, Stephen McKay wrote: > >> MD5 1bfe5d8702009e431bad3862b79f5d95 foo > >> MD5 8687c409c3cc95464d49a7c4dff71e73 quux > > BSDI use: > > 1bfe5d8702009e431bad3862b79f5d95 foo > 8687c409c3cc95464d49a7c4dff71e73 quux I'm certainly willing to change the "-r" format to this. Opinions? -- -- David (obrien@NUXI.com) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Nov 9 8:46:10 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 9FBCF14D5F for ; Tue, 9 Nov 1999 08:45:59 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id RAA23020 for ; Tue, 9 Nov 1999 17:45:54 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id RAA00947 for freebsd-arch@freebsd.org; Tue, 9 Nov 1999 17:45:54 +0100 (MET) Received: from wall.polstra.com (rtrwan160.accessone.com [206.213.115.74]) by hub.freebsd.org (Postfix) with ESMTP id 489BC14CF0; Tue, 9 Nov 1999 08:45:39 -0800 (PST) (envelope-from jdp@polstra.com) Received: from vashon.polstra.com (vashon.polstra.com [206.213.73.13]) by wall.polstra.com (8.9.3/8.9.3) with ESMTP id IAA06025; Tue, 9 Nov 1999 08:45:38 -0800 (PST) (envelope-from jdp@polstra.com) From: John Polstra Received: (from jdp@localhost) by vashon.polstra.com (8.9.3/8.9.1) id IAA07670; Tue, 9 Nov 1999 08:45:38 -0800 (PST) (envelope-from jdp@polstra.com) Date: Tue, 9 Nov 1999 08:45:38 -0800 (PST) Message-Id: <199911091645.IAA07670@vashon.polstra.com> To: obrien@freebsd.org Subject: Re: new MD5 option In-Reply-To: <19991108231948.B6537@dragon.nuxi.com> References: <19991107223137.C29186@dragon.nuxi.com> <199911080813.SAA25572@nymph.detir.qld.gov.au> <19991108231948.B6537@dragon.nuxi.com> Organization: Polstra & Co., Seattle, WA Cc: arch@freebsd.org Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In article <19991108231948.B6537@dragon.nuxi.com>, David O'Brien wrote: > On Mon, Nov 08, 1999 at 06:13:16PM +1000, Stephen McKay wrote: > > BSDI use: > > > > 1bfe5d8702009e431bad3862b79f5d95 foo > > 8687c409c3cc95464d49a7c4dff71e73 quux > > I'm certainly willing to change the "-r" format to this. > Opinions? Yes. Two pieces of information, two fields. Filename last, just like the historical "sum" utility. John -- John Polstra jdp@polstra.com John D. Polstra & Co., Inc. Seattle, Washington USA "No matter how cynical I get, I just can't keep up." -- Nora Ephron To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Nov 9 16:25:22 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 33CB1151DA for ; Tue, 9 Nov 1999 16:25:11 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id BAA27461 for ; Wed, 10 Nov 1999 01:25:10 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id BAA04250 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 01:25:09 +0100 (MET) Received: from nomis.simon-shapiro.org (nomis.simon-shapiro.org [209.86.126.163]) by hub.freebsd.org (Postfix) with SMTP id E743B151DA for ; Tue, 9 Nov 1999 16:24:56 -0800 (PST) (envelope-from shimon@simon-shapiro.org) Received: (qmail 9135 invoked from network); 10 Nov 1999 00:24:51 -0000 Received: from localhost.simon-shapiro.org (HELO simon-shapiro.org) (127.0.0.1) by localhost.simon-shapiro.org with SMTP; 10 Nov 1999 00:24:51 -0000 Message-ID: <3828BB53.DD482CD2@simon-shapiro.org> Date: Tue, 09 Nov 1999 19:24:51 -0500 From: Simon Shapiro Organization: Simon's Garage X-Mailer: Mozilla 4.6 [en] (X11; I; FreeBSD 3.3-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: freebsd-arch@freebsd.org Subject: I/O Evaluation Questions (Long but interesting!) Content-Type: text/plain; charset= Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hi Y'll Am working on a paper describing some I/O findings. The full text will appear on my web page some day soon, but I encountered some questions I would like input on. If this forum is the wrong one, tell me. All my measurements were done (unless otherwise specified) on a Dell PowerEdge 1300/600. Running UP RELENG_3 (SMP crashes on this box with some bizarre errors - irrelevant here). Disk subsystem is the DPT PM3755UW2 hooked up to 3 disk shelves; 2 with 7,200 ST39173LC (Ultra2 Barracuda), and 1 with 10,000 ST39102LC (Ultra2 Cheetah, with which I have no end of trouble - irrelevant). Very Relevant: The firmware on these controllers does NOT perform any READ caching (there is a way to trigger it on, but beats me what it is). It will cache WRITE operations, but the main use of the cache is to manage RAID-5 parity. Keep this in mind as you look at raw I/O results as these are with no cache in the kernel. Thus READ operations are not cached at all, except on block devices. 2 Main programs were used; dd: Used for sequential I/O, single user evaluation st.d: Used for random I/O, multi-user and stress loading. Base Line: # dd if=/dev/zero of=/dev/null bs=128k count=2048 2048+0 records in 2048+0 records out 268435456 bytes transferred in 0.603104 secs (445,089,832 bytes/sec) # st.d -f /dev/zero -s 1024m -i 100 & Throughput = 467893.33 I/O ops/Sec Bandwidth = 1827.7083 MB/Sec [ The above means to run st.d in (default) read mode, use the file /dev/zero for input, use (by default) random seeks), force a file size of 1GB (st.d normally finds sizes by itself), run 100 concurrent instances ] This establishes, that the given system can, from software point of view, perform close to half million I/O operations per second (ops/Sec), and generate and move almost 2GB of data per second, on a single CPU. Next, was to compile the KERNEL with the option I2O_IS_DEV_NULL. This leaves the i2o driver intact, except that it always completes everything successfully, without ever calling the hardware. This excludes any DMA overhead, but all the queue management, locking and other mess still runs. st.d -f /dev/ri2o0 -s 1024m -p 1000000 -i 100 & Throughput = 4043250.39 I/O ops/Sec Bandwidth = 15793.9469 MB/Sec [ The -p 1000000 forces 1 million passes. It is too fast otherwise ] From this we see that the driver consumes somewhat less than 0.25 microsecond per call. The megabytes per second are bogus as no data gets copied (raw device). Tests Run: The full story will be in the above mentioned paper, but essentially, we ran random seek and sequential tests on both raw devices and on block devices. We used either single disks (2GB partition), or a RAID-0 array (15GB partition). Results Highlights: RAW Devices (mostly with 100 workers): Sequential READ: RAID-0 = 47,001,025 bytes/sec (358 128K ops/Sec, 11,456 4K ops/Sec?) RAID-5+0 = 43,308,308 bytes/sec (330 128K ops/Sec, 10,560 4K ops/Sec?) Sequential WRITE: RAID-0 = 34,234,343 bytes/sec (261 128K ops/Sec, 8,352 4K ops/Sec?) RAID-5+0 = 24,624,350 bytes/sec (187 128K ops/Sec, 5,984 4k ops/Sec?) Random READ: RAID-0: = 1,660 ops/Sec RAID-5+0 = 1,943 ops/Sec Random WRITE: RAID-0 = 1,500 ops/Sec RAID-5+0 = 2,397 ops/Sec Block Devices: Sequential READ: RAID-0 = 14,263,937 bytes/sec (108 128K ops/Sec, 3,456 4K ops/Sec?) RAID-5+0 = 14,110,286 bytes/sec (107 128K ops/Sec, 3,425 4K ops/Sec?) Sequential WRITE: RAID-0 = 31,407,413 bytes/sec (958 128K ops/Sec, 30,656 4K ops/Sec?) RAID-5+0 = 33,707,151 bytes/sec (257 128K ops/Sec, 8,224 4K ops/Sec?) Random READ: RAID-0 = 33,578 ops/Sec (300 workers) RAID-5+0 = 42,784 ops/Sec (400 workers) Random WRITE: RAID-0 = 112.68 ops/Sec (200 workers) RAID-5+0 = 112.46 ops/Sec (100 workers) And this, ladies and gentlemen is what I do not understand; Why is random WRITE to a block device about 10-11 times slower than raw device? Actually, sequential read is 1/3 of raw device too. Why? I would _love_ it to be the driver, but the driver has no clue who/what calls it. Besides, the driver contributes less than 0.02% of these numbers. The IOP? It definitely has no clue. One interesting observation; The disks are way to busy for the random block device numbers. Parting Notes: * This is not a CAM problem. The OSM has nothing to do with CAM. * Source code for everything discussed here is at my ftp server, or my web server. * I'd really like to understand this problem. Any help rendered will be greatly appreciated. -- Sincerely Yours, Shimon@Simon-Shapiro.ORG 404.664.6401 Simon Shapiro Unwritten code has no bugs and executes at twice the speed of mouth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Nov 9 19:23:53 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 88EAD14BC4 for ; Tue, 9 Nov 1999 19:23:45 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id EAA01984 for ; Wed, 10 Nov 1999 04:23:39 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id EAA04937 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 04:23:37 +0100 (MET) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id 770E414BC4 for ; Tue, 9 Nov 1999 19:23:23 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from p13-ts5.syd2.zeta.org.au (beefcake.zeta.org.au [203.26.10.12]) by mailman.zeta.org.au (8.8.7/8.8.7) with ESMTP id OAA19800; Wed, 10 Nov 1999 14:28:58 +1100 Date: Wed, 10 Nov 1999 14:22:58 +1100 (EST) From: Bruce Evans X-Sender: bde@alphplex.bde.org To: Simon Shapiro Cc: freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) In-Reply-To: <3828BB53.DD482CD2@simon-shapiro.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > And this, ladies and gentlemen is what I do not understand; > > Why is random WRITE to a block device about 10-11 times > slower than raw device? > Actually, sequential read is 1/3 of raw device too. Why? Block devices have to use a fixed block size. This size is normally BLKDEV_IOSIZE. For historical reasons, BLKDEV_IOSIZE is normally too small. On i386's, it is 2048 in RELENG_3 and 4096 in -current. -current has a sysctl to set the default size. Large i/o's are split up into blocks of the fixed size. Small blocks are very bad for sequential i/o's. They may actually be good for random i/o's if the original i/o's are small. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Nov 9 20:23:11 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 7CA2E15168 for ; Tue, 9 Nov 1999 20:23:08 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id FAA02277 for ; Wed, 10 Nov 1999 05:23:08 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id FAA05064 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 05:23:08 +0100 (MET) Received: from nomis.simon-shapiro.org (nomis.simon-shapiro.org [209.86.126.163]) by hub.freebsd.org (Postfix) with SMTP id 6572115168 for ; Tue, 9 Nov 1999 20:22:59 -0800 (PST) (envelope-from shimon@simon-shapiro.org) Received: (qmail 53495 invoked from network); 10 Nov 1999 04:22:58 -0000 Received: from localhost.simon-shapiro.org (HELO simon-shapiro.org) (127.0.0.1) by localhost.simon-shapiro.org with SMTP; 10 Nov 1999 04:22:58 -0000 Message-ID: <3828F322.B62D174E@simon-shapiro.org> Date: Tue, 09 Nov 1999 23:22:58 -0500 From: Simon Shapiro Organization: Simon's Garage X-Mailer: Mozilla 4.6 [en] (X11; I; FreeBSD 3.3-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Bruce Evans Cc: freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) References: Content-Type: text/plain; charset= Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Bruce Evans wrote: > > > And this, ladies and gentlemen is what I do not understand; > > > > Why is random WRITE to a block device about 10-11 times > > slower than raw device? > > Actually, sequential read is 1/3 of raw device too. Why? > > Block devices have to use a fixed block size. This size is > normally BLKDEV_IOSIZE. For historical reasons, BLKDEV_IOSIZE > is normally too small. On i386's, it is 2048 in RELENG_3 and > 4096 in -current. -current has a sysctl to set the default > size. > > Large i/o's are split up into blocks of the fixed size. Small > blocks are very bad for sequential i/o's. They may actually be > good for random i/o's if the original i/o's are small. > > Bruce Thanx Bruce. What I observed, is that ALL block device I/O is happening in 8KB calls, except for the 512 bytes calls for slice and partition management (5 of them, methinks). The 10:1 random write problem may be mine; An ancient and well hidden bug in st.d which made lock unlock calls for every i/o. We run circles around NT in the Random I/O department, but take a beating in the sequential I/O arena; For about the same hardware, they do 98 MB/Sec, I cannot get more than 45. Their test machine has 64bit PCI. Mine does not. -- Sincerely Yours, Shimon@Simon-Shapiro.ORG 404.664.6401 Simon Shapiro Unwritten code has no bugs and executes at twice the speed of mouth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Nov 9 21:39:31 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id D449814DE5 for ; Tue, 9 Nov 1999 21:39:29 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id GAA02646 for ; Wed, 10 Nov 1999 06:39:23 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id GAA05228 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 06:39:23 +0100 (MET) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id 75A6B14DE5 for ; Tue, 9 Nov 1999 21:39:10 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from p13-ts5.syd2.zeta.org.au (beefcake.zeta.org.au [203.26.10.12]) by mailman.zeta.org.au (8.8.7/8.8.7) with ESMTP id QAA30776; Wed, 10 Nov 1999 16:44:52 +1100 Date: Wed, 10 Nov 1999 16:38:51 +1100 (EST) From: Bruce Evans X-Sender: bde@alphplex.bde.org To: Simon Shapiro Cc: freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) In-Reply-To: <3828F322.B62D174E@simon-shapiro.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > > Block devices have to use a fixed block size. This size is > > normally BLKDEV_IOSIZE. For historical reasons, BLKDEV_IOSIZE > What I observed, is that ALL block device I/O is happening > in 8KB calls, except for the 512 bytes calls for slice > and partition management (5 of them, methinks). 8KB is probably for a filesystem-related block size being used instead of BLKDEV_IOSIZE. This happens when the device is a partition on a labeled drive and the label entry says that the partition type is 4.2BSD and the block size is 8192. > The 10:1 random write problem may be mine; An ancient > and well hidden bug in st.d which made lock unlock calls > for every i/o. Drivers can have this bug too :-). The wd driver has it :-(. In RELENG_3, the specfs i/o routines check the label for every block, although this is worse than useless. Checking the label involves calling the driver ioctl routine, and wdioctl() does careful locking for all ioctls although it doesn't need to for most. wd's locking involves waiting until the controller is inactive and this has bad effects on the throughput. > We run circles around NT in the Random I/O department, > but take a beating in the sequential I/O arena; > For about the same hardware, they do 98 MB/Sec, > I cannot get more than 45. I've always though FreeBSD has the opposite problem. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Tue Nov 9 22: 9:31 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id A186A14EFC for ; Tue, 9 Nov 1999 22:09:29 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id HAA02829 for ; Wed, 10 Nov 1999 07:09:28 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id HAA05301 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 07:09:28 +0100 (MET) Received: from nomis.simon-shapiro.org (nomis.simon-shapiro.org [209.86.126.163]) by hub.freebsd.org (Postfix) with SMTP id 16E4614EFC for ; Tue, 9 Nov 1999 22:09:21 -0800 (PST) (envelope-from shimon@simon-shapiro.org) Received: (qmail 54860 invoked from network); 10 Nov 1999 06:09:08 -0000 Received: from localhost.simon-shapiro.org (HELO simon-shapiro.org) (127.0.0.1) by localhost.simon-shapiro.org with SMTP; 10 Nov 1999 06:09:08 -0000 Message-ID: <38290C04.8FC73862@simon-shapiro.org> Date: Wed, 10 Nov 1999 01:09:08 -0500 From: Simon Shapiro Organization: Simon's Garage X-Mailer: Mozilla 4.6 [en] (X11; I; FreeBSD 3.3-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Bruce Evans Cc: freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) References: Content-Type: text/plain; charset= Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Bruce Evans wrote: > > > > Block devices have to use a fixed block size. This size is > > > normally BLKDEV_IOSIZE. For historical reasons, BLKDEV_IOSIZE > > > What I observed, is that ALL block device I/O is happening > > in 8KB calls, except for the 512 bytes calls for slice > > and partition management (5 of them, methinks). > > 8KB is probably for a filesystem-related block size being used > instead of BLKDEV_IOSIZE. This happens when the device is > a partition on a labeled drive and the label entry says that > the partition type is 4.2BSD and the block size is 8192. > > > The 10:1 random write problem may be mine; An ancient > > and well hidden bug in st.d which made lock unlock calls > > for every i/o. > > Drivers can have this bug too :-). The wd driver has it :-(. > In RELENG_3, the specfs i/o routines check the label for every > block, although this is worse than useless. Checking the > label involves calling the driver ioctl routine, and wdioctl() > does careful locking for all ioctls although it doesn't need to > for most. wd's locking involves waiting until the controller > is inactive and this has bad effects on the throughput. > > > We run circles around NT in the Random I/O department, > > but take a beating in the sequential I/O arena; > > For about the same hardware, they do 98 MB/Sec, > > I cannot get more than 45. > > I've always though FreeBSD has the opposite problem. Nope. I am getting 167MB/Sec for random block device. We are almost nine times faster on random WRITE (and I am comparing RAW I/O here to buffered there), twice as fast on random READ (again our RAW vs. their buffered. If you compare our block perfromance to theirs, we are almost fourty times faster. We are on-par on sequential WRITEs; we get 10MB/Sec RAW and 14 block, they report under 20. There must be some nasty trick to their sequential READ; It is 5 times as fast as their sequential WRITE. BTW, I may have stepped on a hornet nest here; panic: getnewbuf: inconsistent EMPTY queue, qindex=2 Another Question to this honorable forum: Other than heavy, multiple process, concurrent sequential and random I/O to the block and raw devices, what good way is there to verify that the OSM is working correctly? Making a filesystem is always successful, and after a panic, it fsck's perfectly, and the panic appears to be in other filesystems. > Bruce -- Sincerely Yours, Shimon@Simon-Shapiro.ORG 404.664.6401 Simon Shapiro Unwritten code has no bugs and executes at twice the speed of mouth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Nov 10 10:13:55 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 3AF8A1527A for ; Wed, 10 Nov 1999 10:13:49 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id TAA13704 for ; Wed, 10 Nov 1999 19:13:47 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id TAA07799 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 19:13:47 +0100 (MET) Received: from mail.tvol.com (mail.wgate.com [38.219.83.4]) by hub.freebsd.org (Postfix) with ESMTP id C608B1525E for ; Wed, 10 Nov 1999 10:13:34 -0800 (PST) (envelope-from rjesup@wgate.com) Received: from jesup.eng.tvol.net (jesup.eng.tvol.net [10.32.2.26]) by mail.tvol.com (8.8.8/8.8.3) with ESMTP id NAA10242 for ; Wed, 10 Nov 1999 13:09:45 -0500 (EST) Reply-To: Randell Jesup To: freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) From: Randell Jesup Date: 10 Nov 1999 13:13:27 -0500 Message-ID: X-Mailer: Gnus v5.6.43/Emacs 20.4 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Simon Shapiro writes: >> > We run circles around NT in the Random I/O department, >> > but take a beating in the sequential I/O arena; >> > For about the same hardware, they do 98 MB/Sec, >> > I cannot get more than 45. >> >> I've always though FreeBSD has the opposite problem. > >Nope. I am getting 167MB/Sec for random block device. >We are almost nine times faster on random WRITE (and I >am comparing RAW I/O here to buffered there), twice as >fast on random READ (again our RAW vs. their buffered. > >If you compare our block perfromance to theirs, we are >almost fourty times faster. > >We are on-par on sequential WRITEs; we get 10MB/Sec RAW and >14 block, they report under 20. > >There must be some nasty trick to their sequential READ; >It is 5 times as fast as their sequential WRITE. It's really hard to comment without knowing exactly what you were testing on each, and whether they really were equivalent tests. First, are you certain they're not caching the data, even if they're not supposed to be? This would be my #1 guess. Second, could they be (for large IO's) transferring directly into user memory, bypassing all buffers (I haven't really been following the discussion; a good trick is to do direct DMA into the destination buffer - it also allows you to use large commands to the drive (less command overhead). Saving a memory-to-memory copy counts at those speeds. Third, they could be doing some severe page table magic and not actually reading most of the data into physical memory until it's accessed. Unlikely, though, and very tricky. (Interesting idea, though - pseudo-mmap.) They also could set up the DMA, and mark the pages in the page table so that you'll fault if you try to access them, and then undo the mark when the IO is done (or as each N pages of the IO is done make those N pages accessible). There are many cute tricks here... What hardware do you have that gives 100MB/s or more??? -- Randell Jesup, Worldgate Communications, ex-Scala, ex-Amiga OS team ('88-94) rjesup@wgate.com CDA II has been passed and signed, sigh. The lawsuit has been filed. Please support the organizations fighting it - ACLU, EFF, CDT, etc. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Nov 10 12: 7: 9 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id DFFF714C8C for ; Wed, 10 Nov 1999 12:06:52 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id VAA14568 for ; Wed, 10 Nov 1999 21:06:49 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id VAA08226 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 21:06:46 +0100 (MET) Received: from relay.nuxi.com (nuxi.cs.ucdavis.edu [169.237.7.38]) by hub.freebsd.org (Postfix) with ESMTP id B902F15337 for ; Wed, 10 Nov 1999 12:02:07 -0800 (PST) (envelope-from obrien@NUXI.com) Received: from dragon.nuxi.com (root@d60-025.leach.ucdavis.edu [169.237.60.25]) by relay.nuxi.com (8.9.3/8.9.3) with ESMTP id MAA57415 for ; Wed, 10 Nov 1999 12:02:05 -0800 (PST) (envelope-from obrien@dragon.nuxi.com) Received: (from obrien@localhost) by dragon.nuxi.com (8.9.3/8.9.1) id MAA61534 for arch@freebsd.org; Wed, 10 Nov 1999 12:02:05 -0800 (PST) (envelope-from obrien) Date: Wed, 10 Nov 1999 12:02:05 -0800 From: "David O'Brien" To: arch@freebsd.org Subject: compat library source organization Message-ID: <19991110120205.E5897@dragon.nuxi.com> Reply-To: obrien@NUXI.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0i X-Operating-System: FreeBSD 4.0-CURRENT Organization: The NUXI BSD group X-PGP-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Keyid: 34F9F9D5 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hi all, We're starting to see the need for compat3x libraries for the Alpha. So the question becomes how should we organize the source tree to deal with this? I'd like to propose src/lib/compat//{compat3x,compat4x,etc..} This would require a repository copy of src/lib/compat/compat3x, but those files aren't tagged, so we can do a repo move vs. copy. Thus there won't be any added bloat to the tree. I propose to leave compat{1x,2{0,1,2}} where they are to reduce repository bloat and to just leave our heritage alone. Opinions? -- -- David (obrien@NUXI.com) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Nov 10 12:30:31 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id DBF9414D71 for ; Wed, 10 Nov 1999 12:30:10 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id VAA14763 for ; Wed, 10 Nov 1999 21:30:08 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id VAA08345 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 21:30:08 +0100 (MET) Received: from alpo.whistle.com (alpo.whistle.com [207.76.204.38]) by hub.freebsd.org (Postfix) with ESMTP id 3113415141 for ; Wed, 10 Nov 1999 12:28:31 -0800 (PST) (envelope-from julian@whistle.com) Received: from current1.whiste.com (current1.whistle.com [207.76.205.22]) by alpo.whistle.com (8.9.1a/8.9.1) with ESMTP id MAA60402 for ; Wed, 10 Nov 1999 12:28:29 -0800 (PST) Date: Wed, 10 Nov 1999 12:28:26 -0800 (PST) From: Julian Elischer To: freebsd-arch@freebsd.org Subject: Threads thread Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Terry, Archie and I are all at a class this week and have very poor connectivity. (I only have connectivity actually IN class) ;-) I was hoping that someone else could make take on the task of making an independent summary of things we discussed. My summary of 'goals' needs to be updated to include the ability to create 'groups' of threads, and for those groups to have the capability of being assigned separate system resource limits (e.g. different priority). This comes in two parts.. from the kernel's point of view, there must be different classes (groups) of KSE's (Kernel Schedulable Entities), But it is completely opaque to the kernel which User level Thread (ULT?) the User level thread scheuler assignes to those KSEs. One would hope that the ULS would co-operate! It seems to me that from the goals we have decided upon we are already seeing some facets of teh implementation becoming clear. Certainly we need Multiple KSEs and the ability of these to be grouped. We also need a ULS (user level Scheduler). What else can we see are becoming 'Inevitable'? Do we allow the thread system to schedule different threads when a thread get's a pagefault? How can we be sure that teh pagefault is not inteh scheduler? etc. etc. Julian (typing on a pretty slow link in class when I should be taking notice of the teacher) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Nov 10 13:54:26 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id CCDFF14E3C for ; Wed, 10 Nov 1999 13:54:21 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id WAA15459 for ; Wed, 10 Nov 1999 22:54:19 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id WAA08688 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 22:54:19 +0100 (MET) Received: from ind.alcatel.com (postal.xylan.com [208.8.0.248]) by hub.freebsd.org (Postfix) with ESMTP id 9FFB314E3C for ; Wed, 10 Nov 1999 13:53:55 -0800 (PST) (envelope-from wes@softweyr.com) Received: from mailhub.xylan.com (mailhub [198.206.181.70]) by ind.alcatel.com (8.9.3+Sun/8.9.1 (ind.alcatel.com 3.0 [OUT])) with SMTP id NAA26729; Wed, 10 Nov 1999 13:53:52 -0800 (PST) Received: from omni.xylan.com by mailhub.xylan.com (SMI-8.6/SMI-SVR4 (mailhub 2.1 [HUB])) id NAA24036; Wed, 10 Nov 1999 13:53:46 -0800 Received: from softweyr.com (dyn0.utah.xylan.com) by omni.xylan.com (4.1/SMI-4.1 (xylan engr [SPOOL])) id AA21130; Wed, 10 Nov 99 13:53:43 PST Message-Id: <3829E967.E53A2960@softweyr.com> Date: Wed, 10 Nov 1999 14:53:43 -0700 From: Wes Peters Organization: Softweyr LLC X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 3.1-RELEASE i386) X-Accept-Language: en Mime-Version: 1.0 To: Julian Elischer Cc: freebsd-arch@freebsd.org Subject: Re: Threads thread References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Julian Elischer wrote: > > Terry, Archie and I are all at a class this week and have very poor > connectivity. (I only have connectivity actually IN class) ;-) I know a company that makes a cool little dial-up router, you could plug it into the phone line in your hotel room and snake an ethernet cable into the next room, too. ;^) -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC wes@softweyr.com http://softweyr.com/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Nov 10 13:58:14 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 398CD14E5E for ; Wed, 10 Nov 1999 13:58:11 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id WAA15515 for ; Wed, 10 Nov 1999 22:58:10 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id WAA08724 for freebsd-arch@freebsd.org; Wed, 10 Nov 1999 22:58:10 +0100 (MET) Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 447B014E5E for ; Wed, 10 Nov 1999 13:58:05 -0800 (PST) (envelope-from bright@wintelcom.net) Received: from localhost (bright@localhost) by fw.wintelcom.net (8.9.3/8.9.3) with ESMTP id OAA29219; Wed, 10 Nov 1999 14:22:56 -0800 (PST) Date: Wed, 10 Nov 1999 14:22:55 -0800 (PST) From: Alfred Perlstein To: Julian Elischer Cc: freebsd-arch@freebsd.org Subject: Re: Threads thread In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Wed, 10 Nov 1999, Julian Elischer wrote: > Do we allow the thread system to schedule different threads when a thread > get's a pagefault? How can we be sure that teh pagefault is not inteh > scheduler? Can't the kernel know because it can look at which stack or part of the stack is being used? ie. special case for the supervisory thread's stack being active? -Alfred To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Nov 10 16:40:50 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id D5F1F14E76 for ; Wed, 10 Nov 1999 16:40:29 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id BAA16699 for ; Thu, 11 Nov 1999 01:40:26 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id BAA09384 for freebsd-arch@freebsd.org; Thu, 11 Nov 1999 01:40:26 +0100 (MET) Received: from plunger.gdeb.com (plunger.gdeb.com [153.11.11.3]) by hub.freebsd.org (Postfix) with ESMTP id 05A8314E76 for ; Wed, 10 Nov 1999 16:39:56 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from orion.caen.gdeb.com ([153.11.109.11]) by plunger.gdeb.com with ESMTP id TAA00805; Wed, 10 Nov 1999 19:35:06 -0500 (EST) Received: from vigrid.com (clcrtr [153.11.109.129]) by orion.caen.gdeb.com (8.9.3/8.9.3) with ESMTP id TAA59008; Wed, 10 Nov 1999 19:37:29 -0500 (EST) (envelope-from eischen@vigrid.com) Message-ID: <382A0FC9.DDF8228F@vigrid.com> Date: Wed, 10 Nov 1999 19:37:29 -0500 From: "Daniel M. Eischen" X-Mailer: Mozilla 4.51 [en] (X11; U; FreeBSD 3.2-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer Cc: freebsd-arch@freebsd.org Subject: Threads goals and implementation References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Julian wanted an updated set of goals and to start defining the implementation details that are becoming clear. -------------------------------------------------------------- 1/ Multiple independent 'threads of control' within a single process at user level. The most basic quality of threads. 2/ Ability to simultaneously schedule M threads over N Processors, and have min(M,N) threads simultaneously executing. 2A/ ability to tune and control the above.. 3/ Just because one thread blocks, doesn't mean that the others can't keep running. 4/ All threads in a processs see the same address space (exactly). 5/ All threads in a process share the same system resources. 6/ (contentious) Multiple theads should be bound to within the resource limits of the single process (see also 13) 7/ Some well documeted scheme exists for handling signals and other async events. 8/ Exit/shutdown protocol is well defined. 9/ There exists a set of primatives that allow threads to influence the in-process scheduling between themselves. 9A/ e.g. 'per thread' Thread scheduling classes. 10/ Quick access to curthread and thread specific data. 11/ A method to ask a thread blocked in the kernel to wake up and back out (similar to present 'signals'). 12/ Processorr affinity for threads. 13/ Ability to create thread groups that can be assigned separate system resource limits (e.g. priority, quantum). ---- possible userland implementation goals----- 1/ A libpthread that can be linked with libc. 2/ Libc needs to change so that library functions and system calls used internal to the library do not use the externally visible cancellable equivalents. ------------- Meta-goals ------------- We should keep our eyes on: *) scalability *) performance *) ability to support features required by standards based threads. *) ability to support features of those therad packages we select as needed. --------------------------- Implementation: 1/ User-level thread scheduler that can perform user-level thread (ULT) context switches. The ULT that is running on a KSE is completely opaque to the kernel. 2/ The kernel provides KSEs to a process in which ULTs may be run. Multiple KSEs may be provided to a process in order to achieve simultaneous SMP executions and to allow ULTs to obtain separate system resource limits. 2A/ Some code to manage assigments of KSEs to processes and CPUs will be needed. 3/ Kernel structures (proc, sleep queues, run queues?) will need to change to support KSEs and multiple threads per process being blocked in the kernel. It seems that struct proc should be analyzed to classify components as being relevant to the process, KSEs, or "context being blocked in the kernel". --------------------------- Questions: 1/ Do we allow the thread system to schedule different threads when a thread gets a pagefault? How can we be sure that the pagefault is not in the scheduler? --------------------------- References (posted again to provide a more complete set) Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism http://www.freebsd.org/~deischen/p95-anderson.pdf FreeBSD-current mail archives with a 'rfork thread create' implementation http://www.freebsd.org/cgi/getmsg.cgi?fetch=25000+30231+/usr/local/www/db/text/1999/freebsd-current/19990321.freebsd-current LinuxThreads http://lt.tar.com/ Adding Scheduler Activations to Mach 3.0 ftp://ftp.cs.washington.edu/tr/1992/08/UW-CSE-92-08-03.PS.Z User-Level Threads and Interprocess Communication ftp://ftp.cs.washington.edu/tr/1993/02/UW-CSE-93-02-03.PS.Z Tools and Techniques for Building Fast Portable Threads Packages ftp://ftp.cs.washington.edu/tr/1993/05/UW-CSE-93-05-06.PS.Z Efficient Support for Fine-Grain Parallelism ftp://ftp.cs.arizona.edu/reports/1993/TR93-13a.ps Performance Experiments for the Filaments Package ftp://ftp.cs.arizona.edu/reports/1993/TR93-26.ps Space-efficient Scheduling for Parallel, Multithreaded Computations http://reports-archive.adm.cs.cmu.edu/anon/1999/CMU-CS-99-119.ps Chorus: ftp://ftp.chorus.fr/pub/chorus-reports/ ftp://ftp.chorus.fr/pub/chorus-reports/CS-TR-91-7.ps.Z ftp://ftp.chorus.fr/pub/chorus-reports/CS-TR-89-37.ps.Z MACH: ftp://ftp.cs.cmu.edu/project/mach/doc/published/Rcs.ps ftp://ftp.cs.cmu.edu/project/mach/doc/unpublished/sched.concur.ps ftp://ftp.cs.cmu.edu/project/mach/doc/unpublished/threads.ps ftp://ftp.cs.cmu.edu/project/mach/doc/published/cont_threads.ps ftp://ftp.cs.cmu.edu/project/mach/doc/published/cmultithread.ps DEC: http://www.crl.research.digital.com/publications/techreports/techreports.html http://www.digital.com/info/DTJF03/DTJF03SC.TXT ftp://ftp.crl.research.digital.com/pub/dec/Alpha/alpha-osf1-call-std-v1_3.ps Solaris: http://suncom.bilkent.edu.tr/workshop/sig/threads/usenix.html http://www.sun.com/smcc/solaris-migration/docs/courses/threadsHTML/contents.html --------------------------- What we have at the moment: John Birrell's excelent work on user-level threading (libc_r), based originally on Chris provenzo's threading code has given us quite a bit of mileage so far, but it's starting to run out of steam with new requirements being more certain about kernel support requirements. It is notable that we already support Linux kernel threads on FreeBSD better than we support a native threads model. This is because we support the 'clone' system call through our rfork() code, and that is their basis for threading. As is common for this group of people, we have not adopted the Linux approach because it is considered to be 'too simplistic', assigning a separate kernel schedulable process to run each thread. Having said that, Amancio Hasty at one stage wrote a set of threading primitives to allow Kafe to run on FreeBSD using this scheme of threading, and Richard Seaman has a port of the Linuxthreads code to freeBSD at his website, http://lt.tar.com/ . This represents a useful piece of work and though it is presently not working on -current, hopefully this will be fixed soon. I believe there may be problems with the new signal stuff though I have not tested it myself. I also reference the following email in the archives: http://www.freebsd.org/cgi/getmsg.cgi?fetch=25000+30231+/usr/local/www/db/text/1999/freebsd-current/19990321.freebsd-current Peter dufault also has some work on scheduling that may be slightly relevent. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Wed Nov 10 16:42: 3 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id D561814E76 for ; Wed, 10 Nov 1999 16:41:54 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id BAA16710 for ; Thu, 11 Nov 1999 01:41:51 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id BAA09406 for freebsd-arch@freebsd.org; Thu, 11 Nov 1999 01:41:50 +0100 (MET) Received: from plunger.gdeb.com (plunger.gdeb.com [153.11.11.3]) by hub.freebsd.org (Postfix) with ESMTP id 1455114E76 for ; Wed, 10 Nov 1999 16:41:34 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from orion.caen.gdeb.com ([153.11.109.11]) by plunger.gdeb.com with ESMTP id TAA00818; Wed, 10 Nov 1999 19:36:44 -0500 (EST) Received: from vigrid.com (clcrtr [153.11.109.129]) by orion.caen.gdeb.com (8.9.3/8.9.3) with ESMTP id TAA59014; Wed, 10 Nov 1999 19:39:04 -0500 (EST) (envelope-from eischen@vigrid.com) Message-ID: <382A1028.639EDBC1@vigrid.com> Date: Wed, 10 Nov 1999 19:39:04 -0500 From: "Daniel M. Eischen" X-Mailer: Mozilla 4.51 [en] (X11; U; FreeBSD 3.2-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer , freebsd-arch@freebsd.org Subject: Re: Threads goals and implementation References: <382A0FC9.DDF8228F@vigrid.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Daniel M. Eischen" wrote: > Questions: > > 1/ Do we allow the thread system to schedule different threads when a thread > gets a pagefault? How can we be sure that the pagefault is not in the > scheduler? I think we can. I think the SA paper covers this by checking for a second outstanding page fault at the same address. In this case, SAs are delayed until the page fault clears. Dan Eischen eischen@vigrid.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 2:55:54 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id C8B3914C36 for ; Thu, 11 Nov 1999 02:55:46 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id LAA23029 for ; Thu, 11 Nov 1999 11:55:45 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id LAA11128 for freebsd-arch@freebsd.org; Thu, 11 Nov 1999 11:55:44 +0100 (MET) Received: from mail.scc.nl (node1374.a2000.nl [62.108.19.116]) by hub.freebsd.org (Postfix) with ESMTP id DFA2B14C36 for ; Thu, 11 Nov 1999 02:55:35 -0800 (PST) (envelope-from freebsd-arch@scc.nl) Received: (from daemon@localhost) by mail.scc.nl (8.9.3/8.9.3) id LAA89932 for arch@FreeBSD.org; Thu, 11 Nov 1999 11:36:17 +0100 (CET) (envelope-from freebsd-arch@scc.nl) Received: from GATEWAY by dwarf.hq.scc.nl with netnews for arch@FreeBSD.org (arch@FreeBSD.org) To: arch@freebsd.org Date: Thu, 11 Nov 1999 11:36:09 +0100 From: Marcel Moolenaar Message-ID: <382A9C19.5B48D728@scc.nl> Organization: SCC vof Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit References: <19991110120205.E5897@dragon.nuxi.com> Subject: Re: compat library source organization Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG David O'Brien wrote: > I'd like to propose src/lib/compat//{compat3x,compat4x,etc..} from bsd.subdir.mk: if test -d ${.CURDIR}/$${entry}.${MACHINE}; then \ "===> ${DIRPRFX}$${entry}.${MACHINE}"; \ edir=$${entry}.${MACHINE}; \ cd ${.CURDIR}/$${edir}; \ else \ ${ECHODIR} "===> ${DIRPRFX}$$entry"; \ edir=$${entry}; \ cd ${.CURDIR}/$${edir}; \ fi; \ ergo: src/lib/compat.i386/... and src/lib/compat.alpha/... is the proper way to do it, if you want to have seperate directories at all. -- Marcel Moolenaar mailto:marcel@scc.nl SCC Internetworking & Databases http://www.scc.nl/ The FreeBSD project mailto:marcel@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 4:48:47 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 7AAFC14D94 for ; Thu, 11 Nov 1999 04:48:44 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id NAA25202 for ; Thu, 11 Nov 1999 13:48:43 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id NAA11467 for freebsd-arch@freebsd.org; Thu, 11 Nov 1999 13:48:42 +0100 (MET) Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by hub.freebsd.org (Postfix) with ESMTP id 534B514D94 for ; Thu, 11 Nov 1999 04:48:31 -0800 (PST) (envelope-from michael.schuster@germany.sun.com) Received: from emuc05-home.Germany.Sun.COM ([129.157.51.10]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id EAA12493 for ; Thu, 11 Nov 1999 04:48:28 -0800 (PST) Received: from germany.sun.com (hacker [129.157.167.97]) by emuc05-home.Germany.Sun.COM (8.8.8+Sun/8.8.8/ENSMAIL,v1.7) with ESMTP id NAA23509 for ; Thu, 11 Nov 1999 13:48:22 +0100 (MET) Message-ID: <382ABB1A.CB959F01@germany.sun.com> Date: Thu, 11 Nov 1999 13:48:26 +0100 From: Michael Schuster - TSC SunOS Germany Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.7 [en] (X11; I; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation References: <382A0FC9.DDF8228F@vigrid.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Hi all, I guess I'm being a bit facetious (sp?) here, nevertheless: "Daniel M. Eischen" wrote: > -------------------------------------------------------------- > > 1/ Multiple independent 'threads of control' within a single process > at user level. The most basic quality of threads. > > 2/ Ability to simultaneously schedule M threads over N Processors, > and have min(M,N) threads simultaneously executing. > 2A/ ability to tune and control the above.. > > 3/ Just because one thread blocks, doesn't mean that > the others can't keep running. > > 4/ All threads in a processs see the same address space (exactly). > > 5/ All threads in a process share the same system resources. IMO, this is a contradiction to 13/ and 9/ > 6/ (contentious) Multiple theads should be bound to within the resource > limits of the single process (see also 13) This is implied by 5/ (I see resource limits as "resource" as well - "meta" resource, if you will). > 7/ Some well documeted scheme exists for handling signals and other async > events. > > 8/ Exit/shutdown protocol is well defined. > > 9/ There exists a set of primatives that allow threads to influence the > in-process scheduling between themselves. > 9A/ e.g. 'per thread' Thread scheduling classes. scheduling class is an attribute of a thread, therefore a resource -> ergo contradiction to 5/ & 6/ > 10/ Quick access to curthread and thread specific data. see my suggestion to 5/ below; otherwise, this belongs to implementation issues ("quick access"). > 11/ A method to ask a thread blocked in the kernel to wake up and back > out (similar to present 'signals'). > > 12/ Processorr affinity for threads. > > 13/ Ability to create thread groups that can be assigned separate system > resource limits (e.g. priority, quantum). see my comment to 6/ my suggestion: 5/ All threads in a process share the same resources by default with the following possible exceptions 5a/ the (limits for) the following resources can be set on a per-thread basis: priority, quantum, scheduling class.. (your favourite here) 5b/ thread-specific data such as curthread, thread stack, etc. and do away with 6/, 9/, 10/ and 13/ regards Michael PS: Of course, one can always argue that this is just a question of how you view and order information in your mind ... no contradiction here... -- Michael Schuster / Michael.Schuster@germany.sun.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 9:51:23 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 68F0D14CA2 for ; Thu, 11 Nov 1999 09:51:15 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id SAA29697 for ; Thu, 11 Nov 1999 18:51:13 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id SAA12616 for freebsd-arch@freebsd.org; Thu, 11 Nov 1999 18:51:13 +0100 (MET) Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 9195D14CA2 for ; Thu, 11 Nov 1999 09:51:05 -0800 (PST) (envelope-from eischen@vigrid.com) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id MAA27528; Thu, 11 Nov 1999 12:49:47 -0500 (EST) Date: Thu, 11 Nov 1999 12:49:41 -0500 (EST) From: Daniel Eischen To: Michael Schuster - TSC SunOS Germany Cc: "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation In-Reply-To: <382ABB1A.CB959F01@germany.sun.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Thu, 11 Nov 1999, Michael Schuster - TSC SunOS Germany wrote: > > 5/ All threads in a process share the same system resources. > > IMO, this is a contradiction to 13/ and 9/ Perhaps 5 and 13 should be combined into one. > > 6/ (contentious) Multiple theads should be bound to within the resource > > limits of the single process (see also 13) > > This is implied by 5/ (I see resource limits as "resource" as well - > "meta" resource, if you will). OK > > 9/ There exists a set of primatives that allow threads to influence the > > in-process scheduling between themselves. > > 9A/ e.g. 'per thread' Thread scheduling classes. > > scheduling class is an attribute of a thread, therefore a resource -> > ergo contradiction to 5/ & 6/ This doesn't have to imply scheduling across all threads in the system. It does state "in-process" scheduling, so I don't think that 9A is meant to include system scheduling classes. > > 10/ Quick access to curthread and thread specific data. > > see my suggestion to 5/ below; otherwise, this belongs to implementation > issues ("quick access"). > > > 11/ A method to ask a thread blocked in the kernel to wake up and back > > out (similar to present 'signals'). > > > > 12/ Processorr affinity for threads. > > > > 13/ Ability to create thread groups that can be assigned separate system > > resource limits (e.g. priority, quantum). > > see my comment to 6/ > > my suggestion: > > 5/ All threads in a process share the same resources by default with > the following possible exceptions > 5a/ the (limits for) the following resources can be set on a > per-thread basis: priority, quantum, scheduling class.. (your favourite > here) > 5b/ thread-specific data such as curthread, thread stack, etc. > > and do away with 6/, 9/, 10/ and 13/ Makes sense to me, though I think that "quick access to TSD and current thread" is a goal. Dan Eischen eischen@vigrid.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 15:36:48 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 2E85214F70 for ; Thu, 11 Nov 1999 15:36:39 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id AAA02836 for ; Fri, 12 Nov 1999 00:36:37 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id AAA14368 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 00:36:36 +0100 (MET) Received: from nomis.simon-shapiro.org (nomis.simon-shapiro.org [209.86.126.163]) by hub.freebsd.org (Postfix) with SMTP id 9D02F14F4D for ; Thu, 11 Nov 1999 15:36:26 -0800 (PST) (envelope-from shimon@simon-shapiro.org) Received: (qmail 38325 invoked from network); 11 Nov 1999 23:36:25 -0000 Received: from localhost.simon-shapiro.org (HELO simon-shapiro.org) (127.0.0.1) by localhost.simon-shapiro.org with SMTP; 11 Nov 1999 23:36:25 -0000 Message-ID: <382B52F9.2C6D1E00@simon-shapiro.org> Date: Thu, 11 Nov 1999 18:36:25 -0500 From: Simon Shapiro Organization: Simon's Garage X-Mailer: Mozilla 4.6 [en] (X11; I; FreeBSD 3.3-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: Randell Jesup Cc: freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) References: Content-Type: text/plain; charset= Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Randell Jesup wrote: > > Simon Shapiro writes: > >> > We run circles around NT in the Random I/O department, > >> > but take a beating in the sequential I/O arena; > >> > For about the same hardware, they do 98 MB/Sec, > >> > I cannot get more than 45. > >> > >> I've always though FreeBSD has the opposite problem. > > > >Nope. I am getting 167MB/Sec for random block device. > >We are almost nine times faster on random WRITE (and I > >am comparing RAW I/O here to buffered there), twice as > >fast on random READ (again our RAW vs. their buffered. > > > >If you compare our block perfromance to theirs, we are > >almost fourty times faster. > > > >We are on-par on sequential WRITEs; we get 10MB/Sec RAW and > >14 block, they report under 20. > > > >There must be some nasty trick to their sequential READ; > >It is 5 times as fast as their sequential WRITE. > > It's really hard to comment without knowing exactly what > you were testing on each, and whether they really were equivalent tests. Of course you are right, but, at times, I come across info that is not to be quoted at the source. > First, are you certain they're not caching the data, even > if they're not supposed to be? This would be my #1 guess. AKAIK, you really have to be an NT kernel hacker to get raw i/o. > Second, could they be (for large IO's) transferring directly > into user memory, bypassing all buffers (I haven't really been following > the discussion; a good trick is to do direct DMA into the destination > buffer - it also allows you to use large commands to the drive (less > command overhead). Saving a memory-to-memory copy counts at those speeds. This happens in FreeBSD on raw I/O. I belive some work was done to do that on block i.o too (something to do with zero copy in vm... > Third, they could be doing some severe page table magic and not > actually reading most of the data into physical memory until it's accessed. Hard to do with the i2o protocol, as you give the IOP a S/G list and have no control from that point on... > Unlikely, though, and very tricky. (Interesting idea, though - > pseudo-mmap.) They also could set up the DMA, and mark the pages in the > page table so that you'll fault if you try to access them, and then undo > the mark when the IO is done (or as each N pages of the IO is done make > those N pages accessible). There are many cute tricks here... > > What hardware do you have that gives 100MB/s or more??? (bragging corner: 167 read, 138 write :-) DPT PM3755U2B with 256MB of ECC cache in a Dell PowerEdge 1300/600. FreeBSD RELENG_3, single CPU running. > -- > Randell Jesup, Worldgate Communications, ex-Scala, ex-Amiga OS team ('88-94) > rjesup@wgate.com > CDA II has been passed and signed, sigh. The lawsuit has been filed. Please > support the organizations fighting it - ACLU, EFF, CDT, etc. > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message -- Sincerely Yours, Shimon@Simon-Shapiro.ORG 404.664.6401 Simon Shapiro Unwritten code has no bugs and executes at twice the speed of mouth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 15:42:30 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id E15B914F4D for ; Thu, 11 Nov 1999 15:42:27 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id AAA02867 for ; Fri, 12 Nov 1999 00:42:26 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id AAA14404 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 00:42:26 +0100 (MET) Received: from feral.com (feral.com [192.67.166.1]) by hub.freebsd.org (Postfix) with ESMTP id 057A814F61 for ; Thu, 11 Nov 1999 15:42:18 -0800 (PST) (envelope-from mjacob@feral.com) Received: from semuta.feral.com (semuta [192.67.166.70]) by feral.com (8.8.7/8.8.7) with ESMTP id PAA23715; Thu, 11 Nov 1999 15:42:21 -0800 Date: Thu, 11 Nov 1999 15:42:21 -0800 (PST) From: Matthew Jacob Reply-To: mjacob@feral.com To: Simon Shapiro Cc: Randell Jesup , freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) In-Reply-To: <382B52F9.2C6D1E00@simon-shapiro.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > AKAIK, you really have to be an NT kernel hacker to get raw i/o. > No. It's easily available from userland because all of the NT drivers still install DOS compatibility names when they configure themselves: // open the 'D:' drive HANDLE fh = CreateFile(("\\\\.\\PHYSICALDRIVE3", 0, FILE_SHARE_READ|FILE_SHARE_WRITE, 0, OPEN_EXISTING, 0, NULL); if (fh == INVALID_HANDLE_VALUE) { return (-1); } .... To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 17:17:23 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 962A314C0E for ; Thu, 11 Nov 1999 17:17:19 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id CAA03728 for ; Fri, 12 Nov 1999 02:17:17 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id CAA14973 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 02:17:15 +0100 (MET) Received: from panzer.kdm.org (panzer.kdm.org [216.160.178.169]) by hub.freebsd.org (Postfix) with ESMTP id B439314DF4 for ; Thu, 11 Nov 1999 17:16:55 -0800 (PST) (envelope-from ken@panzer.kdm.org) Received: (from ken@localhost) by panzer.kdm.org (8.9.3/8.9.1) id SAA30871; Thu, 11 Nov 1999 18:16:16 -0700 (MST) (envelope-from ken) Message-Id: <199911120116.SAA30871@panzer.kdm.org> Subject: Re: I/O Evaluation Questions (Long but interesting!) In-Reply-To: <382B52F9.2C6D1E00@simon-shapiro.org> from Simon Shapiro at "Nov 11, 1999 06:36:25 pm" To: shimon@simon-shapiro.org (Simon Shapiro) Date: Thu, 11 Nov 1999 18:16:16 -0700 (MST) Cc: rjesup@wgate.com (Randell Jesup), freebsd-arch@freebsd.org From: "Kenneth D. Merry" X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG [ Simon: the "charset = " (i.e. nothing) line your mail makes my mailer barf. You may want to adjust your character set. ] Simon Shapiro wrote... > Randell Jesup wrote: > > Second, could they be (for large IO's) transferring directly > > into user memory, bypassing all buffers (I haven't really been following > > the discussion; a good trick is to do direct DMA into the destination > > buffer - it also allows you to use large commands to the drive (less > > command overhead). Saving a memory-to-memory copy counts at those speeds. > > This happens in FreeBSD on raw I/O. I belive some work was done > to do that on block i.o too (something to do with zero copy > in vm... I think you may get this behavior if you turn on the ENABLE_VFS_IOOPT option, and then 'sysctl -w vfs.ioopt=1'. It only works for page-sized and page-aligned buffers, though. (see sys/ufs/ufs/ufs_readwrite.c) I'm not sure how often that kicks in in normal operation. Someone else might know. > > Unlikely, though, and very tricky. (Interesting idea, though - > > pseudo-mmap.) They also could set up the DMA, and mark the pages in the > > page table so that you'll fault if you try to access them, and then undo > > the mark when the IO is done (or as each N pages of the IO is done make > > those N pages accessible). There are many cute tricks here... > > > > What hardware do you have that gives 100MB/s or more??? > > (bragging corner: 167 read, 138 write :-) DPT PM3755U2B with > 256MB of ECC cache in a Dell PowerEdge 1300/600. > FreeBSD RELENG_3, single CPU running. How can you get speeds like that with just a 32-bit PCI bus? The specs for the PowerEdge 1300 say it has 5 32-bit PCI slots: http://www.dell.com/us/en/biz/products/spec_wrkgp_1300_servers.htm Ken -- Kenneth Merry ken@kdm.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 19:40:34 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 3D13E14DD2 for ; Thu, 11 Nov 1999 19:40:29 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id EAA09353 for ; Fri, 12 Nov 1999 04:40:24 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id EAA15670 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 04:40:22 +0100 (MET) Received: from nomis.simon-shapiro.org (nomis.simon-shapiro.org [209.86.126.163]) by hub.freebsd.org (Postfix) with SMTP id DA7ED14F47 for ; Thu, 11 Nov 1999 19:40:09 -0800 (PST) (envelope-from shimon@simon-shapiro.org) Received: (qmail 41817 invoked from network); 12 Nov 1999 03:40:08 -0000 Received: from localhost.simon-shapiro.org (HELO simon-shapiro.org) (127.0.0.1) by localhost.simon-shapiro.org with SMTP; 12 Nov 1999 03:40:08 -0000 Message-ID: <382B8C18.DD84F967@simon-shapiro.org> Date: Thu, 11 Nov 1999 22:40:08 -0500 From: Simon Shapiro Organization: Simon's Garage X-Mailer: Mozilla 4.6 [en] (X11; I; FreeBSD 3.3-STABLE i386) X-Accept-Language: en MIME-Version: 1.0 To: "Kenneth D. Merry" Cc: Randell Jesup , freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) References: <199911120116.SAA30871@panzer.kdm.org> Content-Type: text/plain; charset= Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Kenneth D. Merry" wrote: > > [ Simon: the "charset = " (i.e. nothing) line your mail makes my mailer > barf. You may want to adjust your character set. ] [ Am using Netscape Messenger. Know not how to do that (no relevant preference found :-( ] > Simon Shapiro wrote... > > Randell Jesup wrote: > > > Second, could they be (for large IO's) transferring directly > > > into user memory, bypassing all buffers (I haven't really been following > > > the discussion; a good trick is to do direct DMA into the destination > > > buffer - it also allows you to use large commands to the drive (less > > > command overhead). Saving a memory-to-memory copy counts at those speeds. > > > > This happens in FreeBSD on raw I/O. I belive some work was done > > to do that on block i.o too (something to do with zero copy > > in vm... > > I think you may get this behavior if you turn on the ENABLE_VFS_IOOPT > option, and then 'sysctl -w vfs.ioopt=1'. It only works for page-sized and > page-aligned buffers, though. (see sys/ufs/ufs/ufs_readwrite.c) I'll try that. Thanx. > I'm not sure how often that kicks in in normal operation. Someone else > might know. > > > > Unlikely, though, and very tricky. (Interesting idea, though - > > > pseudo-mmap.) They also could set up the DMA, and mark the pages in the > > > page table so that you'll fault if you try to access them, and then undo > > > the mark when the IO is done (or as each N pages of the IO is done make > > > those N pages accessible). There are many cute tricks here... > > > > > > What hardware do you have that gives 100MB/s or more??? > > > > (bragging corner: 167 read, 138 write :-) DPT PM3755U2B with > > 256MB of ECC cache in a Dell PowerEdge 1300/600. > > FreeBSD RELENG_3, single CPU running. > > How can you get speeds like that with just a 32-bit PCI bus? The specs for > the PowerEdge 1300 say it has 5 32-bit PCI slots: These numbers are for block devices. The kernel obviously caches some of this. I should look next time at emory usage; The machine has 1GB of memory. The dataset is about 15GB per array. I am getting about 120MB/Sec form the PCI bus. Raw disks perfromance is totally throttled by physics; We are running at about 200% of Seagate specs. I am running into some strange situations. Perhaps some light can be shed; ... int result; ... result = tsleep(lock, PRII2O | PCATCH, "i2olck", wait); switch (result) { ... } The above runs correctly. switch(result = tsleep(lock, PRII2O | PCATCH, "i2olck", wait)) { This line crashes with an invalid pointer in tsleep(). > http://www.dell.com/us/en/biz/products/spec_wrkgp_1300_servers.htm > > Ken > -- > Kenneth Merry > ken@kdm.org -- Sincerely Yours, Shimon@Simon-Shapiro.ORG 404.664.6401 Simon Shapiro Unwritten code has no bugs and executes at twice the speed of mouth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 20: 1:13 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id B89F914F3B for ; Thu, 11 Nov 1999 20:01:11 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id FAA09446 for ; Fri, 12 Nov 1999 05:01:10 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id FAA15746 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 05:01:10 +0100 (MET) Received: from panzer.kdm.org (panzer.kdm.org [216.160.178.169]) by hub.freebsd.org (Postfix) with ESMTP id DB21A14F3B for ; Thu, 11 Nov 1999 20:01:01 -0800 (PST) (envelope-from ken@panzer.kdm.org) Received: (from ken@localhost) by panzer.kdm.org (8.9.3/8.9.1) id VAA31700; Thu, 11 Nov 1999 21:00:42 -0700 (MST) (envelope-from ken) Message-Id: <199911120400.VAA31700@panzer.kdm.org> Subject: Re: I/O Evaluation Questions (Long but interesting!) In-Reply-To: <382B8C18.DD84F967@simon-shapiro.org> from Simon Shapiro at "Nov 11, 1999 10:40:08 pm" To: shimon@simon-shapiro.org (Simon Shapiro) Date: Thu, 11 Nov 1999 21:00:42 -0700 (MST) Cc: rjesup@wgate.com (Randell Jesup), freebsd-arch@freebsd.org From: "Kenneth D. Merry" X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Simon Shapiro wrote... > "Kenneth D. Merry" wrote: > > > > [ Simon: the "charset = " (i.e. nothing) line your mail makes my mailer > > barf. You may want to adjust your character set. ] > [ Am using Netscape Messenger. Know not how to do that > (no relevant preference found :-( ] My best guess is, go to: Edit -> Preferences -> Navigator -> Languages And make sure you at least have English defined there. Also, go to: View -> Character Set And make sure you've got Western (ISO-8559-1) defined. > > Simon Shapiro wrote... > > > Randell Jesup wrote: > > > > Unlikely, though, and very tricky. (Interesting idea, though - > > > > pseudo-mmap.) They also could set up the DMA, and mark the pages in the > > > > page table so that you'll fault if you try to access them, and then undo > > > > the mark when the IO is done (or as each N pages of the IO is done make > > > > those N pages accessible). There are many cute tricks here... > > > > > > > > What hardware do you have that gives 100MB/s or more??? > > > > > > (bragging corner: 167 read, 138 write :-) DPT PM3755U2B with > > > 256MB of ECC cache in a Dell PowerEdge 1300/600. > > > FreeBSD RELENG_3, single CPU running. > > > > How can you get speeds like that with just a 32-bit PCI bus? The specs for > > the PowerEdge 1300 say it has 5 32-bit PCI slots: > > These numbers are for block devices. The kernel obviously > caches some of this. I should look next time at emory usage; > The machine has 1GB of memory. The dataset is about 15GB per > array. Is that for random or sequential I/O? With sequential I/O, you would probably blow away any caching effects. With random I/O, though, you might get significant help from the cache, especially with that much RAM. > I am getting about 120MB/Sec form the PCI > bus. I can believe that. > Raw disks perfromance is totally throttled by physics; > We are running at about 200% of Seagate specs. How can you run at 200% of the spec? Most of the time disk manufacturers are even a little optimistic about their high end performance. > I am running into some strange situations. Perhaps some > light can be shed; Sorry, no clue there. Ken -- Kenneth Merry ken@kdm.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 20:16:50 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 2DF61154AD for ; Thu, 11 Nov 1999 20:16:36 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id FAA09543 for ; Fri, 12 Nov 1999 05:16:34 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id FAA15823 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 05:16:34 +0100 (MET) Received: from nomis.simon-shapiro.org (nomis.simon-shapiro.org [209.86.126.163]) by hub.freebsd.org (Postfix) with SMTP id 95923154C7 for ; Thu, 11 Nov 1999 20:15:01 -0800 (PST) (envelope-from shimon@simon-shapiro.org) Received: (qmail 42339 invoked from network); 12 Nov 1999 04:15:00 -0000 Received: from localhost.simon-shapiro.org (HELO simon-shapiro.org) (127.0.0.1) by localhost.simon-shapiro.org with SMTP; 12 Nov 1999 04:15:00 -0000 Message-ID: <382B9444.44600C3@simon-shapiro.org> Date: Thu, 11 Nov 1999 23:15:00 -0500 From: Simon Shapiro Organization: Simon's Garage X-Mailer: Mozilla 4.6 [en] (X11; I; FreeBSD 3.3-STABLE i386) X-Accept-Language: en-US MIME-Version: 1.0 To: "Kenneth D. Merry" Cc: Randell Jesup , freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) References: <199911120400.VAA31700@panzer.kdm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Kenneth D. Merry" wrote: > > Simon Shapiro wrote... > > "Kenneth D. Merry" wrote: > > > > > > [ Simon: the "charset = " (i.e. nothing) line your mail makes my mailer > > > barf. You may want to adjust your character set. ] > > [ Am using Netscape Messenger. Know not how to do that > > (no relevant preference found :-( ] > > My best guess is, go to: > > Edit -> Preferences -> Navigator -> Languages > > And make sure you at least have English defined there. Also, go to: > > View -> Character Set > > And make sure you've got Western (ISO-8559-1) defined. How's that? (sorry for the spam...) > > > Simon Shapiro wrote... > > > > Randell Jesup wrote: > > > > > Unlikely, though, and very tricky. (Interesting idea, though - > > > > > pseudo-mmap.) They also could set up the DMA, and mark the pages in the > > > > > page table so that you'll fault if you try to access them, and then undo > > > > > the mark when the IO is done (or as each N pages of the IO is done make > > > > > those N pages accessible). There are many cute tricks here... > > > > > > > > > > What hardware do you have that gives 100MB/s or more??? > > > > > > > > (bragging corner: 167 read, 138 write :-) DPT PM3755U2B with > > > > 256MB of ECC cache in a Dell PowerEdge 1300/600. > > > > FreeBSD RELENG_3, single CPU running. > > > > > > How can you get speeds like that with just a 32-bit PCI bus? The specs for > > > the PowerEdge 1300 say it has 5 32-bit PCI slots: > > > > These numbers are for block devices. The kernel obviously > > caches some of this. I should look next time at emory usage; > > The machine has 1GB of memory. The dataset is about 15GB per > > array. > > Is that for random or sequential I/O? With sequential I/O, you would > probably blow away any caching effects. With random I/O, though, you might > get significant help from the cache, especially with that much RAM. Random, of course. To stay architectually minded, please consider these thoughts: Increasing the workers load in this test increases measured throughput (which is to be expected). However, past about 400 concurrent workers, performance declines rapidly. At about 600 the system simply goes nuts. Processes exit or hang solidly without any warnings. Must be some resources to be increased. How is the ftp.cdrom.com kernel configured? This may help me. > > I am getting about 120MB/Sec form the PCI > > bus. > > I can believe that. > > > Raw disks perfromance is totally throttled by physics; > > We are running at about 200% of Seagate specs. > > How can you run at 200% of the spec? Most of the time disk manufacturers > are even a little optimistic about their high end performance. I suspect caching on the disk. I also know the DPT firmware, while claiming not to do READ caching, does some very interesting things with sorting, queuing, tagging, etc. This is worth the difference. More or less. BTW, I am not looking at claimed benchmarks from the mfgs. I am looking at what tends to be accurately reported; Sek times, internal transfer rates, data sheets timing specs, etc. > > I am running into some strange situations. Perhaps some > > light can be shed; > > Sorry, no clue there. :-( Holds me and a bunch others back. > Ken > -- > Kenneth Merry > ken@kdm.org -- Sincerely Yours, Shimon@Simon-Shapiro.ORG 404.664.6401 Simon Shapiro Unwritten code has no bugs and executes at twice the speed of mouth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 20:44:48 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 9128E14E0E for ; Thu, 11 Nov 1999 20:44:46 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id FAA09726 for ; Fri, 12 Nov 1999 05:44:44 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id FAA15908 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 05:44:38 +0100 (MET) Received: from panzer.kdm.org (panzer.kdm.org [216.160.178.169]) by hub.freebsd.org (Postfix) with ESMTP id 8E81C14E0E for ; Thu, 11 Nov 1999 20:44:28 -0800 (PST) (envelope-from ken@panzer.kdm.org) Received: (from ken@localhost) by panzer.kdm.org (8.9.3/8.9.1) id VAA32051; Thu, 11 Nov 1999 21:44:18 -0700 (MST) (envelope-from ken) Message-Id: <199911120444.VAA32051@panzer.kdm.org> Subject: Re: I/O Evaluation Questions (Long but interesting!) In-Reply-To: <382B9444.44600C3@simon-shapiro.org> from Simon Shapiro at "Nov 11, 1999 11:15:00 pm" To: shimon@simon-shapiro.org (Simon Shapiro) Date: Thu, 11 Nov 1999 21:44:18 -0700 (MST) Cc: rjesup@wgate.com (Randell Jesup), freebsd-arch@freebsd.org From: "Kenneth D. Merry" X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Simon Shapiro wrote... > "Kenneth D. Merry" wrote: > > > > Simon Shapiro wrote... > > > "Kenneth D. Merry" wrote: > > > > > > > > [ Simon: the "charset = " (i.e. nothing) line your mail makes my mailer > > > > barf. You may want to adjust your character set. ] > > > [ Am using Netscape Messenger. Know not how to do that > > > (no relevant preference found :-( ] > > > > My best guess is, go to: > > > > Edit -> Preferences -> Navigator -> Languages > > > > And make sure you at least have English defined there. Also, go to: > > > > View -> Character Set > > > > And make sure you've got Western (ISO-8559-1) defined. > > How's that? (sorry for the spam...) Much better, thanks. > > > > How can you get speeds like that with just a 32-bit PCI bus? The specs for > > > > the PowerEdge 1300 say it has 5 32-bit PCI slots: > > > > > > These numbers are for block devices. The kernel obviously > > > caches some of this. I should look next time at emory usage; > > > The machine has 1GB of memory. The dataset is about 15GB per > > > array. > > > > Is that for random or sequential I/O? With sequential I/O, you would > > probably blow away any caching effects. With random I/O, though, you might > > get significant help from the cache, especially with that much RAM. > > Random, of course. Okay, that fits the results. > To stay architectually minded, please consider these thoughts: > > Increasing the workers load in this test increases measured > throughput (which is to be expected). However, past about > 400 concurrent workers, performance declines rapidly. > At about 600 the system simply goes nuts. Processes exit > or hang solidly without any warnings. > Must be some resources to be increased. How is the > ftp.cdrom.com kernel configured? This may help me. wcarchive's configuration might help somewhat (whatever it is), but it is operating with a very different load than the one you're using. It has ~~5000 users, and pushes out I think somewhere in the neighborhood of 100-150Mbits/sec of data. (DG would know for sure.) And it's almost all reads. You're pushing 130-170Mbytes/sec of data, which is about 8 times more, with a fraction of the processes. You may be running into context switch overhead, or who knows what else. The hangs, though, are not good. > > > Raw disks perfromance is totally throttled by physics; > > > We are running at about 200% of Seagate specs. > > > > How can you run at 200% of the spec? Most of the time disk manufacturers > > are even a little optimistic about their high end performance. > > I suspect caching on the disk. I also know the DPT > firmware, while claiming not to do READ caching, does some > very interesting things with sorting, queuing, tagging, etc. > This is worth the difference. More or less. > > BTW, I am not looking at claimed benchmarks from the mfgs. > I am looking at what tends to be accurately reported; > Sek times, internal transfer rates, data sheets timing > specs, etc. I've found that the transfer rates are sometimes accurate. For instance, I've got an IBM Ultrastar 9ZX, which IBM claims can do 10-17MB/sec: http://www.storage.ibm.com/techsup/hddtech/fedspd.pdf That's about right, from what I've seen. The low end may even be a little lower than the actual performance. Another thing you can do is benchmark one disk, and then compare that with the throughput you get from the array. It could be that the combination of the DPT controller's 256MB cache and fancy queueing, and your 1GB of RAM is causing the amazingly fast disk speeds. Ken -- Kenneth Merry ken@kdm.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 21:18:11 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 69B9F14FED for ; Thu, 11 Nov 1999 21:18:08 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id GAA09945 for ; Fri, 12 Nov 1999 06:18:07 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id GAA16039 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 06:18:07 +0100 (MET) Received: from nomis.simon-shapiro.org (nomis.simon-shapiro.org [209.86.126.163]) by hub.freebsd.org (Postfix) with SMTP id 2E3FB14FED for ; Thu, 11 Nov 1999 21:17:58 -0800 (PST) (envelope-from shimon@simon-shapiro.org) Received: (qmail 43261 invoked from network); 12 Nov 1999 05:17:56 -0000 Received: from localhost.simon-shapiro.org (HELO simon-shapiro.org) (127.0.0.1) by localhost.simon-shapiro.org with SMTP; 12 Nov 1999 05:17:56 -0000 Message-ID: <382BA304.EE2F0D66@simon-shapiro.org> Date: Fri, 12 Nov 1999 00:17:56 -0500 From: Simon Shapiro Organization: Simon's Garage X-Mailer: Mozilla 4.6 [en] (X11; I; FreeBSD 3.3-STABLE i386) X-Accept-Language: en-US MIME-Version: 1.0 To: "Kenneth D. Merry" Cc: Randell Jesup , freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) References: <199911120444.VAA32051@panzer.kdm.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG "Kenneth D. Merry" wrote: > > Simon Shapiro wrote... > > "Kenneth D. Merry" wrote: > > > > > > Simon Shapiro wrote... > > > > "Kenneth D. Merry" wrote: > > > > > > > > > > [ Simon: the "charset = " (i.e. nothing) line your mail makes my mailer > > > > > barf. You may want to adjust your character set. ] > > > > [ Am using Netscape Messenger. Know not how to do that > > > > (no relevant preference found :-( ] > > > > > > My best guess is, go to: > > > > > > Edit -> Preferences -> Navigator -> Languages > > > > > > And make sure you at least have English defined there. Also, go to: > > > > > > View -> Character Set > > > > > > And make sure you've got Western (ISO-8559-1) defined. > > > > How's that? (sorry for the spam...) > > Much better, thanks. > > > > > > How can you get speeds like that with just a 32-bit PCI bus? The specs for > > > > > the PowerEdge 1300 say it has 5 32-bit PCI slots: > > > > > > > > These numbers are for block devices. The kernel obviously > > > > caches some of this. I should look next time at emory usage; > > > > The machine has 1GB of memory. The dataset is about 15GB per > > > > array. > > > > > > Is that for random or sequential I/O? With sequential I/O, you would > > > probably blow away any caching effects. With random I/O, though, you might > > > get significant help from the cache, especially with that much RAM. > > > > Random, of course. > > Okay, that fits the results. > > > To stay architectually minded, please consider these thoughts: > > > > Increasing the workers load in this test increases measured > > throughput (which is to be expected). However, past about > > 400 concurrent workers, performance declines rapidly. > > At about 600 the system simply goes nuts. Processes exit > > or hang solidly without any warnings. > > Must be some resources to be increased. How is the > > ftp.cdrom.com kernel configured? This may help me. > > wcarchive's configuration might help somewhat (whatever it is), but it is > operating with a very different load than the one you're using. It has > ~~5000 users, and pushes out I think somewhere in the neighborhood of > 100-150Mbits/sec of data. (DG would know for sure.) And it's almost all > reads. > > You're pushing 130-170Mbytes/sec of data, which is about 8 times more, with > a fraction of the processes. > > You may be running into context switch overhead, or who knows what else. > The hangs, though, are not good. > > > > > Raw disks perfromance is totally throttled by physics; > > > > We are running at about 200% of Seagate specs. > > > > > > How can you run at 200% of the spec? Most of the time disk manufacturers > > > are even a little optimistic about their high end performance. > > > > I suspect caching on the disk. I also know the DPT > > firmware, while claiming not to do READ caching, does some > > very interesting things with sorting, queuing, tagging, etc. > > This is worth the difference. More or less. > > > > BTW, I am not looking at claimed benchmarks from the mfgs. > > I am looking at what tends to be accurately reported; > > Sek times, internal transfer rates, data sheets timing > > specs, etc. > > I've found that the transfer rates are sometimes accurate. For instance, > I've got an IBM Ultrastar 9ZX, which IBM claims can do 10-17MB/sec: > > http://www.storage.ibm.com/techsup/hddtech/fedspd.pdf > > That's about right, from what I've seen. The low end may even be a little > lower than the actual performance. > > Another thing you can do is benchmark one disk, and then compare that with > the throughput you get from the array. Done that. It is exactly on the nose with the specs for sequential and the quoted 200% or less for random. > It could be that the combination of the DPT controller's 256MB cache and > fancy queueing, and your 1GB of RAM is causing the amazingly fast disk speeds. These DPTs seem to be optimal for RAID-5, very good at RAID-0 and nothing exciting for single disks. I have some FC-AL gear on order. What worries me is not the perfromance, but the corruption of the stack that I see. For example, I can run the same 400 processes against the raw device all day and all night without a hitch. Run them against a block device and something bizzare happens; A filesystem get corrupted, the Adaptec driver times out, tsleep segfaults, something. At times I can get the error in the driver, but then it makes no sense either. There are tons of self-checks and state verifications in the code. None trip, or when they do they are as illogical as the null pointer inside tsleep. > Ken > -- > Kenneth Merry > ken@kdm.org > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-arch" in the body of the message -- Sincerely Yours, Shimon@Simon-Shapiro.ORG 404.664.6401 Simon Shapiro Unwritten code has no bugs and executes at twice the speed of mouth To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Thu Nov 11 21:32:37 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 3E7001541B for ; Thu, 11 Nov 1999 21:32:19 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id GAA10063 for ; Fri, 12 Nov 1999 06:32:17 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id GAA16089 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 06:32:17 +0100 (MET) Received: from panzer.kdm.org (panzer.kdm.org [216.160.178.169]) by hub.freebsd.org (Postfix) with ESMTP id 2A38915172 for ; Thu, 11 Nov 1999 21:31:56 -0800 (PST) (envelope-from ken@panzer.kdm.org) Received: (from ken@localhost) by panzer.kdm.org (8.9.3/8.9.1) id WAA32504; Thu, 11 Nov 1999 22:30:22 -0700 (MST) (envelope-from ken) Message-Id: <199911120530.WAA32504@panzer.kdm.org> Subject: Re: I/O Evaluation Questions (Long but interesting!) In-Reply-To: <382BA304.EE2F0D66@simon-shapiro.org> from Simon Shapiro at "Nov 12, 1999 00:17:56 am" To: shimon@simon-shapiro.org (Simon Shapiro) Date: Thu, 11 Nov 1999 22:30:22 -0700 (MST) Cc: rjesup@wgate.com (Randell Jesup), freebsd-arch@freebsd.org From: "Kenneth D. Merry" X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Simon Shapiro wrote... > "Kenneth D. Merry" wrote: > > It could be that the combination of the DPT controller's 256MB cache and > > fancy queueing, and your 1GB of RAM is causing the amazingly fast disk speeds. > > These DPTs seem to be optimal for RAID-5, very good at RAID-0 > and nothing exciting for single disks. I have some FC-AL > gear on order. > > What worries me is not the perfromance, but the corruption > of the stack that I see. > > For example, I can run the same 400 processes against the > raw device all day and all night without a hitch. > Run them against a block device and something bizzare > happens; A filesystem get corrupted, the Adaptec driver > times out, tsleep segfaults, something. At times I can > get the error in the driver, but then it makes no sense > either. There are tons of self-checks and state > verifications in the code. None trip, or when they do > they are as illogical as the null pointer inside tsleep. Well, since you've done a lot of work to try to isolate the problem in your code, but haven't tracked it down, I'd suggest taking your code out of the picture as a variable. Create a CCD or Vinum array, using the same disks on Adaptec controllers. Run the same tests, against the raw and block devices, and see if you get the same sort of weird behavior. If you do, you have solid proof that it's not your code, since your code wasn't in the kernel. If you don't, unfortunately, you don't have solid proof either way. (Since in that case, it could be some set of circumstances that your driver tickles that CCD or Vinum don't.) One other thing to make sure of is that you're running a -stable with Justin's Adaptec driver bug fix from September 20th. It fixed some cases where corruption could happen with Ultra 2 Adaptec controllers. Ken -- Kenneth Merry ken@kdm.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Nov 12 0:20:20 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 6792814C0B for ; Fri, 12 Nov 1999 00:20:16 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id JAA11688 for ; Fri, 12 Nov 1999 09:20:15 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id JAA16510 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 09:20:14 +0100 (MET) Received: from alpo.whistle.com (alpo.whistle.com [207.76.204.38]) by hub.freebsd.org (Postfix) with ESMTP id CA67B14C0B for ; Fri, 12 Nov 1999 00:20:01 -0800 (PST) (envelope-from julian@whistle.com) Received: from current1.whiste.com (current1.whistle.com [207.76.205.22]) by alpo.whistle.com (8.9.1a/8.9.1) with ESMTP id AAA07016; Fri, 12 Nov 1999 00:05:43 -0800 (PST) Date: Fri, 12 Nov 1999 00:05:42 -0800 (PST) From: Julian Elischer To: Michael Schuster - TSC SunOS Germany Cc: "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation In-Reply-To: <382ABB1A.CB959F01@germany.sun.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Thu, 11 Nov 1999, Michael Schuster - TSC SunOS Germany wrote: > > > > 5/ All threads in a process share the same system resources. > > IMO, this is a contradiction to 13/ and 9/ > > > 6/ (contentious) Multiple theads should be bound to within the resource > > limits of the single process (see also 13) > > This is implied by 5/ (I see resource limits as "resource" as well - > "meta" resource, if you will). > > > > > 9/ There exists a set of primatives that allow threads to influence the > > in-process scheduling between themselves. > > 9A/ e.g. 'per thread' Thread scheduling classes. > > scheduling class is an attribute of a thread, therefore a resource -> > ergo contradiction to 5/ & 6/ > > > 10/ Quick access to curthread and thread specific data. > > see my suggestion to 5/ below; otherwise, this belongs to implementation > issues ("quick access"). > > > 13/ Ability to create thread groups that can be assigned separate system > > resource limits (e.g. priority, quantum). > > see my comment to 6/ > > my suggestion: > > 5/ All threads in a process share the same resources by default with > the following possible exceptions > 5a/ the (limits for) the following resources can be set on a > per-thread basis: priority, quantum, scheduling class.. (your favourite > here) > 5b/ thread-specific data such as curthread, thread stack, etc. > > and do away with 6/, 9/, 10/ and 13/ > > PS: Of course, one can always argue that this is just a question of how > you view and order information in your mind ... no contradiction here... > -- Well you do raise some points. here is my altered version: 1/ Multiple independent 'threads of control' within a single process at user level. The most basic quality of threads. 2/ Ability to simultaneously schedule M threads over N Processors, and have min(M,N) threads simultaneously executing. 2A/ ability to tune and control the above.. 3/ Just because one thread blocks, doesn't mean that the others can't keep running. 4/ All threads in a processs see the same address space (exactly). 4A/ All threads in a process share the same system resources, except cpu which is treated specially, and some as yet unspecified thread specific uniqifier. 5/ A process may be able to group threads into classes that have different system priorities. These classes can not have priorities greater than the process itself or a child could achieve, and should be treatable by the system as a separate child process from a scheduling point of view (including limits). 5A/ As a result threaded processes should have no more capability to swamp a system than a regular forking process. 6/ Some well documeted scheme exists for handling signals and other async events. 7/ Exit/shutdown protocol is well defined. 8/ The allocation of user level threads to thread groups is opaque to the kernel. 9/ Quick access to curthread and thread specific data. 10/ A method to ask a thread blocked in the kernel to wake up and back out (similar to present 'signals'). (see 6, 7) ---- possible userland implementation goals----- 1/ A libpthread that can be linked with libc. 2/ Libc needs to change so that library functions and system calls used internal to the library do not use the externally visible cancellable equivalents. 3/ see 8 above. ------------- Meta-goals ------------- We should keep our eyes on: *) scalability *) performance *) ability to support features required by standards based threads. *) ability to support features of those therad packages we select as needed. --------------------------- Implementation: 1/ User-level thread scheduler that can perform user-level thread (ULT) context switches. The ULT that is running on a KSE is completely opaque to the kernel. 2/ The kernel provides KSEs to a process in which ULTs may be run. Multiple KSEs may be provided to a process in order to achieve simultaneous SMP executions and to allow ULTs to obtain separate system resource limits. 2A/ Some code to manage assigments of KSEs to processes and CPUs will be needed. 3/ Kernel structures (proc, sleep queues, run queues?) will need to change to support KSEs and multiple threads per process being blocked in the kernel. It seems that struct proc should be analyzed to classify components as being relevant to the process, KSEs, or "context being blocked in the kernel". Here are my ideas. The terminology is not fixed. A PROCESS is the holder of all resources related to a single address space. The SA paper liked to call this an "Address Space" but I think we want to emphasise it's realtionship to a regular unix process. The PROCESS may 'fork' off several SUBPROCESSES. These share the resources of the process with the exception of their scheduling characteristics. From a scheduling perspective they are the equivalent of child processes. Each SUBPROCESS is allocated one or more KERNEL SCHEDULABLE ENTITIES (KSE's). KSE' may be added to or removed from a SUBPROCESS as needed to prevent it from involuntarily block (except when it's quantum is exceeded)(*) The USER THREAD SCHEDULER assignes thread to KSE's and is aware of all threads entering and exiting the kernel. It can release KSE's and handle the case when a KSE blocks and another is provided as a replacement, It is probably also aware when a KSE is prempted (though only when that PROCESS is next run) (i.e. control returns to the UTS, withthe thread's state available, rather than directly to the thread. The UTS can chose to load that state adn keep running, or allow some other thread to run, depending upon whether there are deadlock or similar ramifications. KSE's from several SUBPROCESSSES may be running simultaniously in an SMP system, but which threads are in them is decided by the UTS. We may need to implement THREAD system calls through a different callgate to allow the different call/return convensions. --------------------------- Questions: 1/ Do we allow the thread system to schedule different threads when a thread gets a pagefault? How can we be sure that the pagefault is not in the scheduler? --------------------------- References (posted again to provide a more complete set) Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism http://www.freebsd.org/~deischen/p95-anderson.pdf FreeBSD-current mail archives with a 'rfork thread create' implementation http://www.freebsd.org/cgi/getmsg.cgi?fetch=25000+30231+/usr/local/www/db/text/1999/freebsd-current/19990321.freebsd-current LinuxThreads http://lt.tar.com/ Adding Scheduler Activations to Mach 3.0 ftp://ftp.cs.washington.edu/tr/1992/08/UW-CSE-92-08-03.PS.Z (* appears unreachable at the moment - JRE *) User-Level Threads and Interprocess Communication ftp://ftp.cs.washington.edu/tr/1993/02/UW-CSE-93-02-03.PS.Z Tools and Techniques for Building Fast Portable Threads Packages ftp://ftp.cs.washington.edu/tr/1993/05/UW-CSE-93-05-06.PS.Z Efficient Support for Fine-Grain Parallelism ftp://ftp.cs.arizona.edu/reports/1993/TR93-13a.ps Performance Experiments for the Filaments Package ftp://ftp.cs.arizona.edu/reports/1993/TR93-26.ps Space-efficient Scheduling for Parallel, Multithreaded Computations http://reports-archive.adm.cs.cmu.edu/anon/1999/CMU-CS-99-119.ps Chorus: ftp://ftp.chorus.fr/pub/chorus-reports/ ftp://ftp.chorus.fr/pub/chorus-reports/CS-TR-91-7.ps.Z ftp://ftp.chorus.fr/pub/chorus-reports/CS-TR-89-37.ps.Z MACH: ftp://ftp.cs.cmu.edu/project/mach/doc/published/Rcs.ps ftp://ftp.cs.cmu.edu/project/mach/doc/unpublished/sched.concur.ps ftp://ftp.cs.cmu.edu/project/mach/doc/unpublished/threads.ps ftp://ftp.cs.cmu.edu/project/mach/doc/published/cont_threads.ps ftp://ftp.cs.cmu.edu/project/mach/doc/published/cmultithread.ps DEC: http://www.crl.research.digital.com/publications/techreports/techreports.html http://www.digital.com/info/DTJF03/DTJF03SC.TXT ftp://ftp.crl.research.digital.com/pub/dec/Alpha/alpha-osf1-call-std-v1_3.ps Solaris: http://suncom.bilkent.edu.tr/workshop/sig/threads/usenix.html http://www.sun.com/smcc/solaris-migration/docs/courses/threadsHTML/contents.html --------------------------- What we have at the moment: John Birrell's excelent work on user-level threading (libc_r), based originally on Chris provenzo's threading code has given us quite a bit of mileage so far, but it's starting to run out of steam with new requirements being more certain about kernel support requirements. It is notable that we already support Linux kernel threads on FreeBSD better than we support a native threads model. This is because we support the 'clone' system call through our rfork() code, and that is their basis for threading. As is common for this group of people, we have not adopted the Linux approach because it is considered to be 'too simplistic', assigning a separate kernel schedulable process to run each thread. Having said that, Amancio Hasty at one stage wrote a set of threading primitives to allow Kafe to run on FreeBSD using this scheme of threading, and Richard Seaman has a port of the Linuxthreads code to freeBSD at his website, http://lt.tar.com/ . This represents a useful piece of work and though it is presently not working on -current, hopefully this will be fixed soon. I believe there may be problems with the new signal stuff though I have not tested it myself. I also reference the following email in the archives: http://www.freebsd.org/cgi/getmsg.cgi?fetch=25000+30231+/usr/local/www/db/text/1999/freebsd-current/19990321.freebsd-current Peter dufault also has some work on scheduling that may be slightly relevent. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Nov 12 0:22:34 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 925E814CEB for ; Fri, 12 Nov 1999 00:22:30 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id JAA11730 for ; Fri, 12 Nov 1999 09:22:29 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id JAA16540 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 09:22:29 +0100 (MET) Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by hub.freebsd.org (Postfix) with ESMTP id E11D614C0B for ; Fri, 12 Nov 1999 00:21:26 -0800 (PST) (envelope-from michael.schuster@germany.sun.com) Received: from emuc05-home.Germany.Sun.COM ([129.157.51.10]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id AAA07443; Fri, 12 Nov 1999 00:21:20 -0800 (PST) Received: from germany.sun.com (hacker [129.157.167.97]) by emuc05-home.Germany.Sun.COM (8.8.8+Sun/8.8.8/ENSMAIL,v1.7) with ESMTP id JAA08497; Fri, 12 Nov 1999 09:21:07 +0100 (MET) Message-ID: <382BCDF7.8689BA4B@germany.sun.com> Date: Fri, 12 Nov 1999 09:21:11 +0100 From: Michael Schuster - TSC SunOS Germany Organization: Sun Microsystems, Inc. X-Mailer: Mozilla 4.7 [en] (X11; I; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Daniel Eischen Cc: "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Daniel Eischen wrote: > On Thu, 11 Nov 1999, Michael Schuster - TSC SunOS Germany wrote: [...] > > > 9/ There exists a set of primatives that allow threads to influence the > > > in-process scheduling between themselves. > > > 9A/ e.g. 'per thread' Thread scheduling classes. > > > > scheduling class is an attribute of a thread, therefore a resource -> > > ergo contradiction to 5/ & 6/ > > This doesn't have to imply scheduling across all threads in the system. > It does state "in-process" scheduling, so I don't think that 9A is > meant to include system scheduling classes. I actually didn't mean to say so, so yes, I agree with you here (am I contradicting myself? I don't really think so :-) > > my suggestion: > > > > 5/ All threads in a process share the same resources by default with > > the following possible exceptions > > 5a/ the (limits for) the following resources can be set on a > > per-thread basis: priority, quantum, scheduling class.. (your favourite > > here) > > 5b/ thread-specific data such as curthread, thread stack, etc. > > > > and do away with 6/, 9/, 10/ and 13/ > > Makes sense to me, though I think that "quick access to TSD and current > thread" is a goal. I meant to say that, I must have lost it somewhere. :-) > Dan Eischen > eischen@vigrid.com cheerio Michael -- Michael Schuster / Michael.Schuster@germany.sun.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Nov 12 0:33:35 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 005A214C09 for ; Fri, 12 Nov 1999 00:33:28 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id JAA11888 for ; Fri, 12 Nov 1999 09:33:27 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id JAA16587 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 09:33:26 +0100 (MET) Received: from ns.mt.sri.com (ns.mt.sri.com [206.127.79.91]) by hub.freebsd.org (Postfix) with ESMTP id 7E74D14CEB for ; Fri, 12 Nov 1999 00:33:09 -0800 (PST) (envelope-from nate@mt.sri.com) Received: from mt.sri.com (rocky.mt.sri.com [206.127.76.100]) by ns.mt.sri.com (8.9.3/8.9.3) with SMTP id BAA05953; Fri, 12 Nov 1999 01:33:03 -0700 (MST) (envelope-from nate@rocky.mt.sri.com) Received: by mt.sri.com (SMI-8.6/SMI-SVR4) id BAA17766; Fri, 12 Nov 1999 01:33:02 -0700 Date: Fri, 12 Nov 1999 01:33:02 -0700 Message-Id: <199911120833.BAA17766@mt.sri.com> From: Nate Williams MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit To: Julian Elischer Cc: Michael Schuster - TSC SunOS Germany , "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation In-Reply-To: References: <382ABB1A.CB959F01@germany.sun.com> X-Mailer: VM 6.34 under 19.16 "Lille" XEmacs Lucid Reply-To: nate@mt.sri.com (Nate Williams) Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG > 10/ A method to ask a thread blocked in the kernel to wake up and back > out (similar to present 'signals'). (see 6, 7) I think this is a good addition. > ---- possible userland implementation goals----- > > 1/ A libpthread that can be linked with libc. > > 2/ Libc needs to change so that library functions and system calls > used internal to the library do not use the externally visible > cancellable equivalents. > 3/ see 8 above. > --------------------------- > > Implementation: > > 1/ User-level thread scheduler that can perform user-level thread (ULT) > context switches. The ULT that is running on a KSE is completely > opaque to the kernel. > > 2/ The kernel provides KSEs to a process in which ULTs may be run. > Multiple KSEs may be provided to a process in order to achieve > simultaneous SMP executions and to allow ULTs to obtain separate > system resource limits. > > 2A/ Some code to manage assigments of KSEs to processes and CPUs will be > needed. > > 3/ Kernel structures (proc, sleep queues, run queues?) will need to > change to support KSEs and multiple threads per process being > blocked in the kernel. It seems that struct proc should be analyzed > to classify components as being relevant to the process, KSEs, or > "context being blocked in the kernel". > 4/ Exceptions may be one way to accomplish thread 'signals', as well as a good way to have the thread 'back out' of kernel context if it is abnormally terminated. > Each SUBPROCESS is allocated one or more KERNEL SCHEDULABLE ENTITIES > (KSE's). KSE' may be added to or removed from a SUBPROCESS as needed to > prevent it from involuntarily block (except when it's quantum is exceeded)(*) How does the kernel re-assign an KSE that corresponds to a blocked thread *IF* the user-land scheduler (or some other active thread) wakes it up abnormally (or otherwise). Does the waking thread lose it's quantum? (Note, I think this is a bad idea in most cases, since what often happens with 'blocked' threads is this.) aquire_resource(); do_something(); wake_waiting_thread(); release_resource(); Note, you must wake the waiting thread while holding onto the resource, so if we wake the waiting thread by giving up our KSE, that thread should still be 'blocked' until we give up the critical resource. So, the 'wakeup' in the normal case is really an event that shouldn't be delivered until *after* the current thread loses it's resource. Nate To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Nov 12 1:47:29 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id BF8D314C96 for ; Fri, 12 Nov 1999 01:47:26 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id KAA13132 for ; Fri, 12 Nov 1999 10:47:26 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id KAA16766 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 10:47:25 +0100 (MET) Received: from flood.ping.uio.no (flood.ping.uio.no [129.240.78.31]) by hub.freebsd.org (Postfix) with ESMTP id 7D3C014C40 for ; Fri, 12 Nov 1999 01:46:25 -0800 (PST) (envelope-from des@flood.ping.uio.no) Received: (from des@localhost) by flood.ping.uio.no (8.9.3/8.9.3) id KAA29030; Fri, 12 Nov 1999 10:46:24 +0100 (CET) (envelope-from des) To: freebsd-arch@freebsd.org Subject: /usr/local From: Dag-Erling Smorgrav Date: 12 Nov 1999 10:46:23 +0100 Message-ID: Lines: 9 User-Agent: Gnus/5.070097 (Pterodactyl Gnus v0.97) Emacs/20.4 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG How would people feel about adding /usr/local/include and /usr/local/lib to gcc's default header and library search paths, respectively? As somebody pointed out in a recent thread on -stable, we already have /usr/local/lib in the default ldconfig_path, so why not in the link-time search path as well? DES -- Dag-Erling Smorgrav - des@flood.ping.uio.no To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Nov 12 2:55:54 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 7AD6214C05 for ; Fri, 12 Nov 1999 02:55:51 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id LAA14252 for ; Fri, 12 Nov 1999 11:55:50 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id LAA26011 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 11:55:50 +0100 (MET) Received: from mail.scc.nl (node1374.a2000.nl [62.108.19.116]) by hub.freebsd.org (Postfix) with ESMTP id BC1E414C05 for ; Fri, 12 Nov 1999 02:55:39 -0800 (PST) (envelope-from freebsd-arch@scc.nl) Received: (from daemon@localhost) by mail.scc.nl (8.9.3/8.9.3) id LAA39347 for arch@FreeBSD.org; Fri, 12 Nov 1999 11:26:42 +0100 (CET) (envelope-from freebsd-arch@scc.nl) Received: from GATEWAY by dwarf.hq.scc.nl with netnews for arch@FreeBSD.org (arch@FreeBSD.org) To: arch@freebsd.org Date: Fri, 12 Nov 1999 11:26:35 +0100 From: Marcel Moolenaar Message-ID: <382BEB5B.EDE3FFC3@scc.nl> Organization: SCC vof Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit References: Subject: Re: /usr/local Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Dag-Erling Smorgrav wrote: > > How would people feel about adding /usr/local/include and > /usr/local/lib to gcc's default header and library search paths, > respectively? As somebody pointed out in a recent thread on -stable, > we already have /usr/local/lib in the default ldconfig_path, so why > not in the link-time search path as well? It can be dangerous. Sometimes people have headers in /usr/local/include that conflict with headers in /usr/include. If /usr/local/include is added as default search path, make world could break. The evidence can be found in spurious postings about some port breakage (rpm for example). Also, having /usr/local/lib in ldconfig_path is not really a precedent for adding /usr/local/include and /usr/local/lib to the standard compile-time search paths. We need (for example) /usr/X11R6/lib in ldconfig_path, but don't want X11 related searches baked into our compiler. If we do it, then it should be done with care. Personally, I don't think it's really necessary... -- Marcel Moolenaar mailto:marcel@scc.nl SCC Internetworking & Databases http://www.scc.nl/ The FreeBSD project mailto:marcel@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Nov 12 3:58:43 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 6818614C96 for ; Fri, 12 Nov 1999 03:58:38 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id MAA15334 for ; Fri, 12 Nov 1999 12:58:37 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id MAA26182 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 12:58:36 +0100 (MET) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id AE35614C96 for ; Fri, 12 Nov 1999 03:58:21 -0800 (PST) (envelope-from bde@zeta.org.au) Received: from p61-ts5.syd2.zeta.org.au (beefcake.zeta.org.au [203.26.10.12]) by mailman.zeta.org.au (8.8.7/8.8.7) with ESMTP id XAA19176; Fri, 12 Nov 1999 23:04:27 +1100 Date: Fri, 12 Nov 1999 22:58:09 +1100 (EST) From: Bruce Evans X-Sender: bde@alphplex.bde.org To: Simon Shapiro Cc: freebsd-arch@freebsd.org Subject: Re: I/O Evaluation Questions (Long but interesting!) In-Reply-To: <38290C04.8FC73862@simon-shapiro.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Wed, 10 Nov 1999, Simon Shapiro wrote: > Bruce Evans wrote: > > > We run circles around NT in the Random I/O department, > > > but take a beating in the sequential I/O arena; > > > For about the same hardware, they do 98 MB/Sec, > > > I cannot get more than 45. > > > > I've always though FreeBSD has the opposite problem. > > Nope. I am getting 167MB/Sec for random block device. I didn't believe this, but later mail explains that it must be due to buffering. > We are almost nine times faster on random WRITE (and I > am comparing RAW I/O here to buffered there), twice as > fast on random READ (again our RAW vs. their buffered. > > If you compare our block perfromance to theirs, we are > almost fourty times faster. This is almost certainly due to our buffer cache actually working for the i/o mix in your test. Having the buffer cache unified with vm helps here by removing arbitrary limits on the effective size of the buffer cache. Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Nov 12 4:51:31 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 87BD414BEC for ; Fri, 12 Nov 1999 04:51:28 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id NAA16159 for ; Fri, 12 Nov 1999 13:51:27 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id NAA26336 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 13:51:27 +0100 (MET) Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 146E614BEC for ; Fri, 12 Nov 1999 04:19:08 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from vigrid.com (pm3-pt33.pcnet.net [206.105.29.107]) by pcnet1.pcnet.com (8.8.7/PCNet) with ESMTP id HAA05359; Fri, 12 Nov 1999 07:17:47 -0500 (EST) Message-ID: <382C0599.1E95429B@vigrid.com> Date: Fri, 12 Nov 1999 07:18:33 -0500 From: "Daniel M. Eischen" X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 4.0-CURRENT i386) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer Cc: Michael Schuster - TSC SunOS Germany , "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Julian Elischer wrote: > Here are my ideas. The terminology is not fixed. > > A PROCESS is the holder of all resources related to a single address > space. The SA paper liked to call this an "Address Space" but I think we > want to emphasise it's realtionship to a regular unix process. > > The PROCESS may 'fork' off several SUBPROCESSES. These share the resources > of the process with the exception of their scheduling characteristics. > >From a scheduling perspective they are the equivalent of child processes. > > Each SUBPROCESS is allocated one or more KERNEL SCHEDULABLE ENTITIES > (KSE's). KSE' may be added to or removed from a SUBPROCESS as needed to > prevent it from involuntarily block (except when it's quantum is exceeded)(*) > > The USER THREAD SCHEDULER assignes thread to KSE's and is aware of all > threads entering and exiting the kernel. No, it is only aware of threads that block and unblock in the kernel. The UTS should not know about _all_ kernel entry points, only those entry points that are cancellable. I suppose that it is possible that the mechanism used to notify the UTS of a thread blocking in the kernel could also include the system call id/number in which the thread was blocked, but it's not necessary from the UTS point of view. > It can release KSE's and handle > the case when a KSE blocks and another is provided as a replacement, It > is probably also aware when a KSE is prempted (though only when that > PROCESS is next run) (i.e. control returns to the UTS, withthe thread's > state available, rather than directly to the thread. The UTS can chose to > load that state adn keep running, or allow some other thread to run, > depending upon whether there are deadlock or similar ramifications. > > KSE's from several SUBPROCESSSES may be running simultaniously in an SMP > system, but which threads are in them is decided by the UTS. > > We may need to implement THREAD system calls through a different callgate > to allow the different call/return convensions. I don't see why this would be necessary. A separate system call to notify the kernel that the process and/or KSEs are using SAs (or whatever we decide on) is sufficient. The UTS would also have to inform the kernel of the user-level routines to use for event notification, so this would probably be sufficient for marking a process as needing different call/return conversions (SAs). I have a question. How do we manage kernel contexts in the kernel? How many threads are allowed to be blocked in the kernel at any one time? Each thread blocked in the kernel uses a resource (probably some structure with a kernel register save area, a TAILQ/LIST entry or two, etc), and this should be constrained in some way. Should pthread_setconcurrency(3) adjust the number of kernel contexts assigned to a process? Dan Eischen eischen@vigrid.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Fri Nov 12 9: 2:23 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 5671314DED for ; Fri, 12 Nov 1999 09:02:10 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id SAA20926 for ; Fri, 12 Nov 1999 18:02:07 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id SAA27426 for freebsd-arch@freebsd.org; Fri, 12 Nov 1999 18:02:06 +0100 (MET) Received: from relay.nuxi.com (nuxi.cs.ucdavis.edu [169.237.7.38]) by hub.freebsd.org (Postfix) with ESMTP id 0425114D57 for ; Fri, 12 Nov 1999 09:01:37 -0800 (PST) (envelope-from obrien@NUXI.com) Received: from dragon.nuxi.com (root@d60-025.leach.ucdavis.edu [169.237.60.25]) by relay.nuxi.com (8.9.3/8.9.3) with ESMTP id JAA67205; Fri, 12 Nov 1999 09:01:36 -0800 (PST) (envelope-from obrien@dragon.nuxi.com) Received: (from obrien@localhost) by dragon.nuxi.com (8.9.3/8.9.1) id JAA87852; Fri, 12 Nov 1999 09:01:36 -0800 (PST) (envelope-from obrien) Date: Fri, 12 Nov 1999 09:01:36 -0800 From: "David O'Brien" To: Dag-Erling Smorgrav Cc: freebsd-arch@freebsd.org Subject: Re: /usr/local Message-ID: <19991112090136.A87828@dragon.nuxi.com> Reply-To: obrien@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0i In-Reply-To: ; from des@flood.ping.uio.no on Fri, Nov 12, 1999 at 10:46:23AM +0100 X-Operating-System: FreeBSD 4.0-CURRENT Organization: The NUXI BSD group X-PGP-Fingerprint: B7 4D 3E E9 11 39 5F A3 90 76 5D 69 58 D9 98 7A X-Pgp-Keyid: 34F9F9D5 Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Fri, Nov 12, 1999 at 10:46:23AM +0100, Dag-Erling Smorgrav wrote: > How would people feel about adding /usr/local/include and > /usr/local/lib to gcc's default header and library search paths, No. This is not PREFIX clean. > we already have /usr/local/lib in the default ldconfig_path, so why > not in the link-time search path as well? Because ldconfig_path is set in a config file (/etc/rc.conf). If the user makes PREFIX point elsewhere, they make a simple edit of rc.conf. Adding -IPREFIX/include to gcc burns this into the binary, and that is not easy for a sysadmin to change. -- -- David (obrien@NUXI.com) To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Nov 13 13:16:50 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 5698C14D1C for ; Sat, 13 Nov 1999 13:16:47 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id WAA11249 for ; Sat, 13 Nov 1999 22:16:46 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id WAA36138 for freebsd-arch@freebsd.org; Sat, 13 Nov 1999 22:16:46 +0100 (MET) Received: from alpo.whistle.com (alpo.whistle.com [207.76.204.38]) by hub.freebsd.org (Postfix) with ESMTP id 1B40414D1C for ; Sat, 13 Nov 1999 13:16:38 -0800 (PST) (envelope-from julian@whistle.com) Received: from current1.whiste.com (current1.whistle.com [207.76.205.22]) by alpo.whistle.com (8.9.1a/8.9.1) with ESMTP id NAA48263; Sat, 13 Nov 1999 13:16:04 -0800 (PST) Date: Sat, 13 Nov 1999 13:16:03 -0800 (PST) From: Julian Elischer To: "Daniel M. Eischen" Cc: Michael Schuster - TSC SunOS Germany , "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation In-Reply-To: <382C0599.1E95429B@vigrid.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Fri, 12 Nov 1999, Daniel M. Eischen wrote: > Julian Elischer wrote: > > Here are my ideas. The terminology is not fixed. > > > > A PROCESS is the holder of all resources related to a single address > > space. The SA paper liked to call this an "Address Space" but I think we > > want to emphasise it's realtionship to a regular unix process. > > > > The PROCESS may 'fork' off several SUBPROCESSES. These share the resources > > of the process with the exception of their scheduling characteristics. > > >From a scheduling perspective they are the equivalent of child processes. > > > > Each SUBPROCESS is allocated one or more KERNEL SCHEDULABLE ENTITIES > > (KSE's). KSE' may be added to or removed from a SUBPROCESS as needed to > > prevent it from involuntarily block (except when it's quantum is exceeded)(*) > > > > The USER THREAD SCHEDULER assignes thread to KSE's and is aware of all > > threads entering and exiting the kernel. > > No, it is only aware of threads that block and unblock in the kernel. > The UTS should not know about _all_ kernel entry points, only those > entry points that are cancellable. I suppose that it is possible that > the mechanism used to notify the UTS of a thread blocking in the kernel > could also include the system call id/number in which the thread was > blocked, but it's not necessary from the UTS point of view. Well, When I say 'aware' I don't mean that it runs a UTS function , but just that it leaves some information behind saying where it has gone.. This allows the scheduler to make better decisions in the case of pre-emption and whether or not it needs more or less KSEs. The point is a large fraction of system calls can block. this is because the copyin() and copyout() functions can pagefault if the destination in userland is not physically present. If a process is pre-empted (Or should I say a sub-process, or KSE?)then when the next sub process for that process is entered, the scheduler should be run immediatly to decide which hread (possibly the pre-empted one) should be continued. This is only possible if the act of resumption can (1) go to the scheduler instead of the thread that was pre-empted, and (2) somehow make the pre-empted state available to the scheduler in the same way as the state for all other suspended threads are available. THere are some questions tht come up from this: 1/ should there be one sub-process for each processor/priority combination? Maybe there is one scheduler 'activation' (hmm) per subprocess? possibly for each subprocess there is only ever one KSE active, but possibly many blocked? Are blocked KSE's assigned to any subprocess, or are they only assigned to the process as a whole? > > > It can release KSE's and handle > > the case when a KSE blocks and another is provided as a replacement, It > > is probably also aware when a KSE is prempted (though only when that > > PROCESS is next run) (i.e. control returns to the UTS, withthe thread's > > state available, rather than directly to the thread. The UTS can chose to > > load that state adn keep running, or allow some other thread to run, > > depending upon whether there are deadlock or similar ramifications. > > > > KSE's from several SUBPROCESSSES may be running simultaniously in an SMP > > system, but which threads are in them is decided by the UTS. > > > > We may need to implement THREAD system calls through a different callgate > > to allow the different call/return convensions. > > I don't see why this would be necessary. A separate system call to > notify the kernel that the process and/or KSEs are using SAs (or whatever > we decide on) is sufficient. The UTS would also have to inform the kernel > of the user-level routines to use for event notification, so this would > probably be sufficient for marking a process as needing different > call/return conversions (SAs). It is probable that we will need a different syscall calling convension to handle teh async nature of the world. If we use a second syscall gate, we can intermix old and new system calls during development. > > I have a question. How do we manage kernel contexts in the kernel? > How many threads are allowed to be blocked in the kernel at any one > time? Each thread blocked in the kernel uses a resource (probably > some structure with a kernel register save area, a TAILQ/LIST entry > or two, etc), and this should be constrained in some way. Should > pthread_setconcurrency(3) adjust the number of kernel contexts > assigned to a process? each KSE has a 'struct thread' which includes a kernel stack and processor context saving space. it is linked back to a struct proc on a circular list. The struct procs describe "SUBPROCESSES" and are in turned grouped in some way to form a 'PROCESS' The more interesting question is "What do we do when we cannot allocate any more of these? (KSEs)" (either through limits or resource shortage). > > Dan Eischen > eischen@vigrid.com > To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Nov 13 15:46:45 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 0830014BCD for ; Sat, 13 Nov 1999 15:46:41 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id AAA12082 for ; Sun, 14 Nov 1999 00:46:37 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id AAA36536 for freebsd-arch@freebsd.org; Sun, 14 Nov 1999 00:46:37 +0100 (MET) Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id E9A4B14BCD for ; Sat, 13 Nov 1999 15:46:00 -0800 (PST) (envelope-from eischen@vigrid.com) Received: from vigrid.com (pm3-pt72.pcnet.net [206.105.29.146]) by pcnet1.pcnet.com (8.8.7/PCNet) with ESMTP id SAA09227; Sat, 13 Nov 1999 18:44:40 -0500 (EST) Message-ID: <382DF821.EE3DE564@vigrid.com> Date: Sat, 13 Nov 1999 18:45:37 -0500 From: "Daniel M. Eischen" X-Mailer: Mozilla 4.5 [en] (X11; I; FreeBSD 4.0-CURRENT i386) X-Accept-Language: en MIME-Version: 1.0 To: Julian Elischer Cc: Michael Schuster - TSC SunOS Germany , "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG Julian Elischer wrote: > > > No, it is only aware of threads that block and unblock in the kernel. > > The UTS should not know about _all_ kernel entry points, only those > > entry points that are cancellable. I suppose that it is possible that > > the mechanism used to notify the UTS of a thread blocking in the kernel > > could also include the system call id/number in which the thread was > > blocked, but it's not necessary from the UTS point of view. > > Well, When I say 'aware' I don't mean that it runs a UTS function , but > just that it leaves some information behind saying where it has gone.. > This allows the scheduler to make better decisions in the case of > pre-emption and whether or not it needs more or less KSEs. The point is a > large fraction of system calls can block. this is because the copyin() and > copyout() functions can pagefault if the destination in userland is not > physically present. Using SA, the UTS is informed when threads block and unblock (including page faults and preemption) in the kernel. As soon as the next "subprocess" is run/resumed, the UTS is notified of these events and can decide which thread to run next. > If a process is pre-empted (Or should I say a sub-process, or KSE?) then > when the next sub process for that process is entered, the scheduler > should be run immediatly to decide which hread (possibly the pre-empted > one) should be continued. This is only possible if the act of resumption > can (1) go to the scheduler instead of the thread that was pre-empted, > and (2) somehow make the pre-empted state available to the scheduler in > the same way as the state for all other suspended threads are available. Exactly the point of SA. An upcall is made to the UTS when the next sub- process is run, and the blocked threads state is passed as a parameter (somewhat simplified) on the stack of the activation. > THere are some questions tht come up from this: > > 1/ should there be one sub-process for each processor/priority > combination? No, it really is up to the UTS and the application to decide (which is then limited by limits/permissions in the kernel). From the user-land point of view, we have the POSIX/SSv2 pthread specs, and our own rtprio(2)/setpriority(2) commands. A userland program would create a PTHREAD_SCOPE_SYSTEM thread which would hopefully create an additional subprocess. At a later point in time, the same thread could change it's process priority, say using rtprio(2), which would change the priority of the subprocess in which it was executing. Additional threads could be created in the same way. > Maybe there is one scheduler 'activation' (hmm) per > subprocess? Reading the "Adding Scheduler Activations to Mach 3.0" paper, they say that there are always exactly as many SAs as there are physical processors. There are more SA stacks than SAs with an added system call to recycle them when they are no longer being used by the UTS. > possibly for each subprocess there is only ever one KSE > active, but possibly many blocked? Are blocked KSE's assigned to any > subprocess, or are they only assigned to the process as a whole? I agree - there is only ever one KSE per subprocess active, and possibly many blocked. I'd think that KSEs are assigned to the process as a whole (while free/blocked). It's up to the UTS to decide what thread to run/resume on what subprocess. A subprocess will need a link to the KSE that it is running, though. > > I don't see why this would be necessary. A separate system call to > > notify the kernel that the process and/or KSEs are using SAs (or whatever > > we decide on) is sufficient. The UTS would also have to inform the kernel > > of the user-level routines to use for event notification, so this would > > probably be sufficient for marking a process as needing different > > call/return conversions (SAs). > > It is probable that we will need a different syscall calling convension > to handle teh async nature of the world. If we use a second syscall gate, > we can intermix old and new system calls during development. OK, I can see how a different syscall gate might be useful during development. > > > > > I have a question. How do we manage kernel contexts in the kernel? > > How many threads are allowed to be blocked in the kernel at any one > > time? Each thread blocked in the kernel uses a resource (probably > > some structure with a kernel register save area, a TAILQ/LIST entry > > or two, etc), and this should be constrained in some way. Should > > pthread_setconcurrency(3) adjust the number of kernel contexts > > assigned to a process? > > each KSE has a 'struct thread' which includes a kernel stack and > processor context saving space. it is linked back to a struct proc on a > circular list. Currently, if I read the source correctly, a procs p_paddr (struct user) includes the kernel register save area and the kernel stack, and is swappable. If we start allowing more kernel contexts (KSEs), do we also want to allow them to be swappable? > The struct procs describe "SUBPROCESSES" and are in turned grouped in some > way to form a 'PROCESS' > > The more interesting question is "What do we do when we > cannot allocate any more of these? (KSEs)" (either through limits or > resource shortage). Tell the UTS when the limit has been reached, then block without SA notification when the limit has been exceeded? Dan Eischen eischen@vigrid.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message From owner-freebsd-arch Sat Nov 13 23:25: 3 1999 Delivered-To: freebsd-arch@freebsd.org Received: from ns1.yes.no (ns1.yes.no [195.204.136.10]) by hub.freebsd.org (Postfix) with ESMTP id 1314114C83 for ; Sat, 13 Nov 1999 23:24:59 -0800 (PST) (envelope-from eivind@bitbox.follo.net) Received: from bitbox.follo.net (bitbox.follo.net [195.204.143.218]) by ns1.yes.no (8.9.3/8.9.3) with ESMTP id IAA15559 for ; Sun, 14 Nov 1999 08:24:54 +0100 (CET) Received: (from eivind@localhost) by bitbox.follo.net (8.8.8/8.8.6) id IAA38789 for freebsd-arch@freebsd.org; Sun, 14 Nov 1999 08:24:54 +0100 (MET) Received: from alpo.whistle.com (alpo.whistle.com [207.76.204.38]) by hub.freebsd.org (Postfix) with ESMTP id D95EB15206 for ; Sat, 13 Nov 1999 23:24:36 -0800 (PST) (envelope-from julian@whistle.com) Received: from current1.whiste.com (current1.whistle.com [207.76.205.22]) by alpo.whistle.com (8.9.1a/8.9.1) with ESMTP id XAA57038; Sat, 13 Nov 1999 23:24:04 -0800 (PST) Date: Sat, 13 Nov 1999 23:24:02 -0800 (PST) From: Julian Elischer To: "Daniel M. Eischen" Cc: Michael Schuster - TSC SunOS Germany , "freebsd-arch@FreeBSD.ORG" Subject: Re: Threads goals and implementation In-Reply-To: <382DF821.EE3DE564@vigrid.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-arch@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG On Sat, 13 Nov 1999, Daniel M. Eischen wrote: > > It is probable that we will need a different syscall calling convension > > to handle teh async nature of the world. If we use a second syscall gate, > > we can intermix old and new system calls during development. > > OK, I can see how a different syscall gate might be useful during > development. more than that.. Old binaries must continue to run, and thus programs linked with libc might continue to use the old syscalls (probably less overhead) while prrograms using libc_r will call the new call-gate with a protocol mor esuited to the new ideas. > > Currently, if I read the source correctly, a procs p_paddr (struct user) > includes the kernel register save area and the kernel stack, and is > swappable. If we start allowing more kernel contexts (KSEs), do we > also want to allow them to be swappable? > possibly. > > The struct procs describe "SUBPROCESSES" and are in turned grouped in some > > way to form a 'PROCESS' > > > > The more interesting question is "What do we do when we > > cannot allocate any more of these? (KSEs)" (either through limits or > > resource shortage). > > Tell the UTS when the limit has been reached, then block without SA > notification when the limit has been exceeded? > It may be reached when some OTHER process uses the last one... so you may not be able to notify.. I guess just blocking is the answer. Julian To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-arch" in the body of the message