From owner-freebsd-smp Sun Dec 3 4:38:12 2000 Delivered-To: freebsd-smp@freebsd.org Received: from correu.andorra.ad (unknown [194.158.78.13]) by hub.freebsd.org (Postfix) with ESMTP id E652C37B401 for ; Sun, 3 Dec 2000 04:20:57 -0800 (PST) Received: from kr8 (m66-6.andorpac.ad [194.158.66.6]) by correu.andorra.ad with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id YBYNVBRH; Sun, 3 Dec 2000 13:20:56 +0100 Message-ID: <00f201c05d24$0e3f2900$94f3fea9@kr8> From: "John Gold" To: Subject: subscribe Date: Sun, 3 Dec 2000 12:24:59 -0000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_00EF_01C05D24.0D975040" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.1 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org This is a multi-part message in MIME format. ------=_NextPart_000_00EF_01C05D24.0D975040 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable ------=_NextPart_000_00EF_01C05D24.0D975040 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

------=_NextPart_000_00EF_01C05D24.0D975040-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Dec 3 6:56: 1 2000 From owner-freebsd-smp@FreeBSD.ORG Sun Dec 3 06:55:59 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id BC5B337B400; Sun, 3 Dec 2000 06:55:57 -0800 (PST) Received: from luanda-08.budapest.interware.hu ([195.70.51.8] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 142aYQ-0002d7-00; Sun, 03 Dec 2000 15:55:51 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A2A5EE3.56E711B8@elischer.org> Date: Sun, 03 Dec 2000 06:55:32 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: brian@freebsd.org, archie@freebsd.org, phk@freebsd.org, smp@freebsd.org Subject: Netgraph and SMP Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I'm trying to figure out the locking needed for SMP in Netgraph. Here are some of the various constraints that I see. (some of theme are not just SMP issues). 1/ can't remove modules that are being used (the base framework) or have living nodes (other netgraph modules). 2/ Cannot remove a node if it is in use, (or in a return stack somewhere) 3/ Cannot unlink an 'edge' if someone is passing data across it. 4/ cannot use edges and nodes that are in the process of being set up or torn down. 5/ Defered deliveries cannot be made to nodes that are being removed. 6/ Nodes cannot be completely removed if there are pending deliveries for them. What locking schemes to otther people use for protecting intricate structures of objects with various interconnections? The buffer cache is one example that comes to mind, as is the routing table. Julian -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Dec 3 7:20:41 2000 From owner-freebsd-smp@FreeBSD.ORG Sun Dec 3 07:20:39 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from correu.andorra.ad (unknown [194.158.78.13]) by hub.freebsd.org (Postfix) with ESMTP id 01AAE37B401 for ; Sun, 3 Dec 2000 07:20:38 -0800 (PST) Received: from KR8 ([194.158.75.69]) by correu.andorra.ad with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id YBYNVCD8; Sun, 3 Dec 2000 16:20:36 +0100 Message-ID: <000201c05d3d$891f6c20$893efea9@kr8> From: "John Gold" To: Subject: dual processor motherboard recommendation for freebsd web/database server Date: Sun, 3 Dec 2000 15:25:24 -0000 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_001B_01C05D3D.41D2DBE0" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 4.72.3110.1 X-MimeOLE: Produced By Microsoft MimeOLE V4.72.3110.3 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org This is a multi-part message in MIME format. ------=_NextPart_000_001B_01C05D3D.41D2DBE0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, could anyone recommend a dual processor motherboard for a web/database = server running freebsd? I would like it to support two intel PIII 800 = mhz cpu's (fcpga socket 370) and will be running a raid disk subsystem. = So far the best choice I have found is a Supermicro 370DL3. Does anyone = have any other suggestions or recommendations, good or bad experiences? many thanks, John Gold gold@kr8.com-nospam ------=_NextPart_000_001B_01C05D3D.41D2DBE0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hi,

could anyone recommend a dual processor motherboard = for a=20 web/database server running freebsd? I would like it to support two = intel PIII=20 800 mhz cpu's (fcpga socket 370) and will be running a raid disk = subsystem.=20 So far the best choice I have found is a Supermicro 370DL3. Does anyone = have any=20 other suggestions or recommendations, good or bad = experiences?

many thanks,

John Gold

gold@kr8.com-nospam

------=_NextPart_000_001B_01C05D3D.41D2DBE0-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sun Dec 3 14:56:17 2000 From owner-freebsd-smp@FreeBSD.ORG Sun Dec 3 14:56:14 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mobile.wemm.org (adsl-64-163-195-99.dsl.snfc21.pacbell.net [64.163.195.99]) by hub.freebsd.org (Postfix) with ESMTP id 4BE7437B400; Sun, 3 Dec 2000 14:56:13 -0800 (PST) Received: from netplex.com.au (localhost [127.0.0.1]) by mobile.wemm.org (8.11.1/8.11.1) with ESMTP id eB3Mu7D37126; Sun, 3 Dec 2000 14:56:11 -0800 (PST) (envelope-from peter@netplex.com.au) Message-Id: <200012032256.eB3Mu7D37126@mobile.wemm.org> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: John Baldwin Cc: Greg Lehey , smp@FreeBSD.ORG, Jake Burkholder Subject: Re: BSD/OS interrupt code In-Reply-To: <200011280601.WAA36325@john.baldwin.cx> Date: Sun, 03 Dec 2000 14:56:07 -0800 From: Peter Wemm Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org John Baldwin wrote: > > On 28-Nov-00 Greg Lehey wrote: > > On Monday, 27 November 2000 at 10:33:05 -0800, John Baldwin wrote: > >> > >> On 26-Nov-00 Jake Burkholder wrote: > >>> Hi, > >>> > >>> If anyone with access to the BSD/OS code is interested, I've written > >>> a little program that runs their interrupt stub code generator in > >>> userland. You can then abort(); and disassemble the stub from > >>> the core dump to look at the code all in one piece. Makes it much > >>> easier to follow. > >>> > >>> In case you haven't looked, their interrupt handlers are generated > >>> by bcopy-ing various blocks of assembler code into an array at > >>> runtime, and then poking in arguments and relocating branches. > >> > >> Hmm. The best way to do this probably is to have stubs with psuedo-reloca tion > >> information. I.e., have 1 stub interrupt handler template for each type ( fast, > >> threaded, fast apic, threaded apic, etc.), then have a "relocation" table on > >> the side that specifies offsets in the code that should receive one of a n umber > >> of things: vector number, vector address, etc. Alternatively, we could st ore > >> all of that info in a struct (gee, we already do with intrhand :)) and jus t > >> have the code always dereference a pointer to that struct and have the > >> relocation entries simply point to places that that pointer should be writ ten. > >> Granted, on sparc this could get much uglier (based on my understanding of > >> relocation address on sparc since an address can be formed across several > >> instructions. Yuck.) > > > > I assume you're talking about doing this at run time, presumably from > > a function like inthand_add. This is a nice, structured way to do > > things. One of the parameters might be the type of interrupt, so > > instead of three stubs for each possible interrupt, we'd only have > > one. Why didn't we do that from the start? Because it's slower. No, > > I haven't tried to prove this, but I'm pretty sure you'd have > > difficulties getting it small enough. > > > > One alternative would be to really create the entire interrupt handler > > stub in inthand_add. It's not as difficult as it probably sounds, but > > in general I agree with Bruce that the saving is probably not worth > > it. > > Doing this would be a pain as Chuck kindly pointed out. One not so difficult step > that can be taken is to make the stubs smaller and have them use more code in > common. For example, you could probably reduce the per-interrupt source port ion > of the stub to just 'pusha; mov $irq_num, %eax; jmp common_code;' in the x86 case. > However, the savings wouldn't be but so great. Nevertheless, it's something I > may play with at some point in time. The sole reason that I can think of to do this is a good one... If we have got more than 24 interrupt sources, there are *EXTENSIVE* modifications to the code required - no less than about 8 or 10 tables, lists, etc needed updating. (I changed this from 24 to 32 on one of my checked out trees, but have not committed it yet). What about machines down the track that have more than that still? One machine that I have at work has something like 10 or 12 PCI slots, with each PCI slot IRQ pin wired to an APIC pin where possible. Mostly it is A and B on the 64 bit/66MHz PCI slots that have two IRQs per slot. Suppose the two APICs had 24 pins, that is 48 interrupt sources.. That is more than I could fit in because of the _apic_imen 32 bit active mask cache. Supporting 48 irq sources, plus potentially the 16 on the PIC in parallel gives us 64 possible IRQ sources, plus the two local cpu interrupt source pins (2*ncpu), that can add up to a few. Now, have a "slow" and "fast" interrupt stub for each and we've easily got up to a theoretical 144 interrupt handler stubs on a 4-cpu system, of which *at least half will be wasted* because a handler is either slow or fast, not both. I guess a multi-gigabyte RAM 4-way or 8-way cpu with multiple APIC's isn't going to notice 72 wasted IRQ stubs, but to make this generic (remember, it cannot be easily ifdefed due to the 32 bit problem), then in order to run in that configuration we'd need to build it into the code for all systems... You could be sure that some magazine will want to do benchmarks on such a high-end system some day and we'd be caught out if we dont support it without extensive code modifications.. ("The benchmarks can only be run on released OS's" etc). Sure, machines of this configuration are not common (yet?), but by the same token, "640k ought to be enough for everybody". Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 9:16:34 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 09:16:32 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from prism.flugsvamp.com (cb58709-a.mdsn1.wi.home.com [24.17.241.9]) by hub.freebsd.org (Postfix) with ESMTP id 4AC7237B400 for ; Mon, 4 Dec 2000 09:16:32 -0800 (PST) Received: (from jlemon@localhost) by prism.flugsvamp.com (8.11.0/8.11.0) id eB4HEF152968; Mon, 4 Dec 2000 11:14:15 -0600 (CST) (envelope-from jlemon) Date: Mon, 4 Dec 2000 11:14:15 -0600 (CST) From: Jonathan Lemon Message-Id: <200012041714.eB4HEF152968@prism.flugsvamp.com> To: julian@elischer.org, smp@freebsd.org Subject: Re: Netgraph and SMP X-Newsgroups: local.mail.freebsd-smp In-Reply-To: Organization: Cc: Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In article you write: > > >I'm trying to figure out the locking needed for SMP in Netgraph. > >Here are some of the various constraints that I see. (some of theme are not just >SMP issues). > >1/ can't remove modules that are being used (the base framework) or have living >nodes (other netgraph modules). >2/ Cannot remove a node if it is in use, (or in a return stack somewhere) >3/ Cannot unlink an 'edge' if someone is passing data across it. >4/ cannot use edges and nodes that are in the process of being set up or torn >down. >5/ Defered deliveries cannot be made to nodes that are being removed. >6/ Nodes cannot be completely removed if there are pending deliveries for them. > >What locking schemes to otther people use for protecting intricate structures of >objects with various interconnections? The buffer cache is one example that >comes to mind, as is the routing table. The routing table (which I'm in the process of locking down) uses reference counts for outstanding references, plus a separate flag to indicate the route is in the routing tree. So it's easier than netgraph, (except trying to untangle the logic). I have patches to make ng_ether_input (and ng_queue_data) MPSAFE from the POV of ether_input. At the moment, it appears that since ether nodes are not removed until the interface is down, there is no race condition. If this wasn't the case, then there would be two possible solutions: 1. place a mutex around the check/call to ng_ether__p, to remove the race condition, (possibly using a ref counter). 2. make ng_ether__p always point to a valid (possibly NOP) subroutine, and use another flag word to indicate whether the routine is should be called or not. Use a timed/delayed delete for freeing the module, to allow existing code to exit. The first solution seems a little too heavyweight, and I haven't extensively checked the second for races with the upper layer code. -- Jonathan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 13:31:22 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 13:31:20 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-176-64.dsl.snfc21.pacbell.net [63.202.176.64]) by hub.freebsd.org (Postfix) with ESMTP id 6E53A37B400; Mon, 4 Dec 2000 13:31:19 -0800 (PST) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.11.0/8.11.1) with ESMTP id eB4Le9F01294; Mon, 4 Dec 2000 13:40:14 -0800 (PST) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200012042140.eB4Le9F01294@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Julian Elischer Cc: brian@freebsd.org, archie@freebsd.org, phk@freebsd.org, smp@freebsd.org Subject: Re: Netgraph and SMP In-reply-to: Your message of "Sun, 03 Dec 2000 06:55:32 PST." <3A2A5EE3.56E711B8@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 04 Dec 2000 13:40:08 -0800 From: Mike Smith Sender: msmith@mass.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org The simplest structure for this is a shared/exclusive lock that supports intention; Terry would have ranted about this. (He would have called it a SIX-lock, I think). In this model, you acquire the lock in 'shared' mode every time you enter Netgraph, and release it when you leave. When you plan to make changes to Netgraph, you get the lock in 'exclusive' mode. 'Intention' comes in here; now that you are trying to get the lock in exclusive mode, your intention is recorded and nobody else can get it in 'shared' mode, so eventually all the users drain out of Netgraph and you get the lock. This may sound simplistic, but given that you don't necessarily make changes to Netgraph very often, this is quite likely more than adequate for now. > I'm trying to figure out the locking needed for SMP in Netgraph. > > Here are some of the various constraints that I see. (some of theme are not just > SMP issues). > > 1/ can't remove modules that are being used (the base framework) or have living > nodes (other netgraph modules). > 2/ Cannot remove a node if it is in use, (or in a return stack somewhere) > 3/ Cannot unlink an 'edge' if someone is passing data across it. > 4/ cannot use edges and nodes that are in the process of being set up or torn > down. > 5/ Defered deliveries cannot be made to nodes that are being removed. > 6/ Nodes cannot be completely removed if there are pending deliveries for them. > > What locking schemes to otther people use for protecting intricate structures of > objects with various interconnections? The buffer cache is one example that > comes to mind, as is the routing table. > > Julian > -- > __--_|\ Julian Elischer > / \ julian@elischer.org > ( OZ ) World tour 2000 > ---> X_.---._/ presently in: Budapest > v > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 14:32:50 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 14:32:45 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from Awfulhak.org (tun.AwfulHak.org [194.242.139.173]) by hub.freebsd.org (Postfix) with ESMTP id F21A937B400; Mon, 4 Dec 2000 14:32:40 -0800 (PST) Received: from hak.lan.Awfulhak.org (root@hak.lan.awfulhak.org [172.16.0.12]) by Awfulhak.org (8.11.1/8.11.1) with ESMTP id eB4MTZM02386; Mon, 4 Dec 2000 22:29:35 GMT (envelope-from brian@hak.lan.Awfulhak.org) Received: from hak.lan.Awfulhak.org (brian@localhost [127.0.0.1]) by hak.lan.Awfulhak.org (8.11.1/8.11.1) with ESMTP id eB4MVaD93192; Mon, 4 Dec 2000 22:31:36 GMT (envelope-from brian@hak.lan.Awfulhak.org) Message-Id: <200012042231.eB4MVaD93192@hak.lan.Awfulhak.org> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: Mike Smith Cc: Julian Elischer , brian@FreeBSD.org, archie@FreeBSD.org, phk@FreeBSD.org, smp@FreeBSD.org, brian@Awfulhak.org Subject: Re: Netgraph and SMP In-Reply-To: Message from Mike Smith of "Mon, 04 Dec 2000 13:40:08 PST." <200012042140.eB4Le9F01294@mass.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 04 Dec 2000 22:31:36 +0000 From: Brian Somers Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > The simplest structure for this is a shared/exclusive lock > that supports intention; Terry would have ranted about this. (He would > have called it a SIX-lock, I think). [.....] > This may sound simplistic, but given that you don't necessarily make > changes to Netgraph very often, this is quite likely more than adequate > for now. Nice, I never realised there were shared/exclusive locks available. I think netgraph nodes would also need to have a ``modevent'' that fails MOD_UNLOAD events if any locks are outstanding. Of course there'll probably be loads of edge-cases :-/ > > I'm trying to figure out the locking needed for SMP in Netgraph. > > > > Here are some of the various constraints that I see. (some of theme are not just > > SMP issues). > > > > 1/ can't remove modules that are being used (the base framework) or have living > > nodes (other netgraph modules). > > 2/ Cannot remove a node if it is in use, (or in a return stack somewhere) > > 3/ Cannot unlink an 'edge' if someone is passing data across it. > > 4/ cannot use edges and nodes that are in the process of being set up or torn > > down. > > 5/ Defered deliveries cannot be made to nodes that are being removed. > > 6/ Nodes cannot be completely removed if there are pending deliveries for them. > > > > What locking schemes to otther people use for protecting intricate structures of > > objects with various interconnections? The buffer cache is one example that > > comes to mind, as is the routing table. > > > > Julian > > -- > > __--_|\ Julian Elischer > > / \ julian@elischer.org > > ( OZ ) World tour 2000 > > ---> X_.---._/ presently in: Budapest > > v > > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > > with "unsubscribe freebsd-smp" in the body of the message > > > > -- > ... every activity meets with opposition, everyone who acts has his > rivals and unfortunately opponents also. But not because people want > to be opponents, rather because the tasks and relationships force > people to take different points of view. [Dr. Fritz Todt] > V I C T O R Y N O T V E N G E A N C E -- Brian Don't _EVER_ lose your sense of humour ! To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 14:41:28 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 14:41:25 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-176-64.dsl.snfc21.pacbell.net [63.202.176.64]) by hub.freebsd.org (Postfix) with ESMTP id 9694737B401; Mon, 4 Dec 2000 14:41:24 -0800 (PST) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.11.0/8.11.1) with ESMTP id eB4Mo7F01738; Mon, 4 Dec 2000 14:50:08 -0800 (PST) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200012042250.eB4Mo7F01738@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Brian Somers Cc: Julian Elischer , brian@FreeBSD.org, archie@FreeBSD.org, phk@FreeBSD.org, smp@FreeBSD.org Subject: Re: Netgraph and SMP In-reply-to: Your message of "Mon, 04 Dec 2000 22:31:36 GMT." <200012042231.eB4MVaD93192@hak.lan.Awfulhak.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 04 Dec 2000 14:50:07 -0800 From: Mike Smith Sender: msmith@mass.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > The simplest structure for this is a shared/exclusive lock > > that supports intention; Terry would have ranted about this. (He would > > have called it a SIX-lock, I think). > [.....] > > This may sound simplistic, but given that you don't necessarily make > > changes to Netgraph very often, this is quite likely more than adequate > > for now. > > Nice, I never realised there were shared/exclusive locks available. > I think netgraph nodes would also need to have a ``modevent'' that > fails MOD_UNLOAD events if any locks are outstanding. Er, no, you just have to acquire the exclusive lock in the MOD_UNLOAD handler. As for the actual availibility of SIX-style locks; I'm fairly sure you can do this with the lockmgr. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 14:46:20 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 14:46:19 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 8DF5E37B400; Mon, 4 Dec 2000 14:46:18 -0800 (PST) Received: from laptop.baldwin.cx (root@dhcp246.osd.bsdi.com [204.216.28.246]) by pike.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id eB4MjsC19112; Mon, 4 Dec 2000 14:45:54 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <200012042250.eB4Mo7F01738@mass.osd.bsdi.com> Date: Mon, 04 Dec 2000 14:46:30 -0800 (PST) From: John Baldwin To: Mike Smith Subject: Re: Netgraph and SMP Cc: smp@FreeBSD.org, phk@FreeBSD.org, archie@FreeBSD.org, brian@FreeBSD.org, Julian Elischer , Brian Somers Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 04-Dec-00 Mike Smith wrote: >> > The simplest structure for this is a shared/exclusive lock >> > that supports intention; Terry would have ranted about this. (He would >> > have called it a SIX-lock, I think). >> [.....] >> > This may sound simplistic, but given that you don't necessarily make >> > changes to Netgraph very often, this is quite likely more than adequate >> > for now. >> >> Nice, I never realised there were shared/exclusive locks available. >> I think netgraph nodes would also need to have a ``modevent'' that >> fails MOD_UNLOAD events if any locks are outstanding. > > Er, no, you just have to acquire the exclusive lock in the MOD_UNLOAD > handler. > > As for the actual availibility of SIX-style locks; I'm fairly sure you > can do this with the lockmgr. Yes. See the allproc_lock as an example. -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 15: 7:52 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 15:07:50 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from crewsoft.com (ns.aenet.net [157.22.214.1]) by hub.freebsd.org (Postfix) with ESMTP id 59EF537B401 for ; Mon, 4 Dec 2000 15:07:50 -0800 (PST) Received: from [63.197.8.222] (HELO wireless-networks.com) by crewsoft.com (CommuniGate Pro SMTP 3.3b5) with ESMTP id 368677 for smp@freebsd.org; Mon, 04 Dec 2000 15:10:01 -0800 Message-ID: <3A2C23ED.14303B3C@wireless-networks.com> Date: Mon, 04 Dec 2000 15:08:29 -0800 From: Cedric Berger X-Mailer: Mozilla 4.73 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: smp@freebsd.org Subject: Re: Netgraph and SMP References: <200012042140.eB4Le9F01294@mass.osd.bsdi.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Mike Smith wrote: > > The simplest structure for this is a shared/exclusive lock > that supports intention; Terry would have ranted about this. (He would > have called it a SIX-lock, I think). > > In this model, you acquire the lock in 'shared' mode every time you enter > Netgraph, and release it when you leave. > > When you plan to make changes to Netgraph, you get the lock in > 'exclusive' mode. 'Intention' comes in here; now that you are trying to > get the lock in exclusive mode, your intention is recorded and nobody > else can get it in 'shared' mode, so eventually all the users drain out > of Netgraph and you get the lock. > How is this SIX-pack lock any different than a standard reader-writer lock? Cedric To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 15:11: 5 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 15:11:03 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from Awfulhak.org (tun.AwfulHak.org [194.242.139.173]) by hub.freebsd.org (Postfix) with ESMTP id 3355537B400; Mon, 4 Dec 2000 15:08:38 -0800 (PST) Received: from hak.lan.Awfulhak.org (root@hak.lan.awfulhak.org [172.16.0.12]) by Awfulhak.org (8.11.1/8.11.1) with ESMTP id eB4N5gM02573; Mon, 4 Dec 2000 23:05:42 GMT (envelope-from brian@hak.lan.Awfulhak.org) Received: from hak.lan.Awfulhak.org (brian@localhost [127.0.0.1]) by hak.lan.Awfulhak.org (8.11.1/8.11.1) with ESMTP id eB4N7gD93751; Mon, 4 Dec 2000 23:07:42 GMT (envelope-from brian@hak.lan.Awfulhak.org) Message-Id: <200012042307.eB4N7gD93751@hak.lan.Awfulhak.org> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: Mike Smith Cc: Brian Somers , Julian Elischer , brian@FreeBSD.org, archie@FreeBSD.org, phk@FreeBSD.org, smp@FreeBSD.org, brian@Awfulhak.org Subject: Re: Netgraph and SMP In-Reply-To: Message from Mike Smith of "Mon, 04 Dec 2000 14:50:07 PST." <200012042250.eB4Mo7F01738@mass.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 04 Dec 2000 23:07:42 +0000 From: Brian Somers Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > The simplest structure for this is a shared/exclusive lock > > > that supports intention; Terry would have ranted about this. (He would > > > have called it a SIX-lock, I think). > > [.....] > > > This may sound simplistic, but given that you don't necessarily make > > > changes to Netgraph very often, this is quite likely more than adequate > > > for now. > > > > Nice, I never realised there were shared/exclusive locks available. > > I think netgraph nodes would also need to have a ``modevent'' that > > fails MOD_UNLOAD events if any locks are outstanding. > > Er, no, you just have to acquire the exclusive lock in the MOD_UNLOAD > handler. Is it desirable to lock up running kldunload(8) ? > -- > ... every activity meets with opposition, everyone who acts has his > rivals and unfortunately opponents also. But not because people want > to be opponents, rather because the tasks and relationships force > people to take different points of view. [Dr. Fritz Todt] > V I C T O R Y N O T V E N G E A N C E -- Brian Don't _EVER_ lose your sense of humour ! To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 15:51: 1 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 15:50:59 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from prism.flugsvamp.com (cb58709-a.mdsn1.wi.home.com [24.17.241.9]) by hub.freebsd.org (Postfix) with ESMTP id 5C06837B400; Mon, 4 Dec 2000 15:50:59 -0800 (PST) Received: (from jlemon@localhost) by prism.flugsvamp.com (8.11.0/8.11.0) id eB4NmeA73673; Mon, 4 Dec 2000 17:48:40 -0600 (CST) (envelope-from jlemon) Date: Mon, 4 Dec 2000 17:48:40 -0600 (CST) From: Jonathan Lemon Message-Id: <200012042348.eB4NmeA73673@prism.flugsvamp.com> To: jhb@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Netgraph and SMP X-Newsgroups: local.mail.freebsd-smp In-Reply-To: References: Organization: Cc: Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In article you write: > >On 04-Dec-00 Mike Smith wrote: >>> > The simplest structure for this is a shared/exclusive lock >>> > that supports intention; Terry would have ranted about this. (He would >>> > have called it a SIX-lock, I think). >>> [.....] >>> > This may sound simplistic, but given that you don't necessarily make >>> > changes to Netgraph very often, this is quite likely more than adequate >>> > for now. >>> >>> Nice, I never realised there were shared/exclusive locks available. >>> I think netgraph nodes would also need to have a ``modevent'' that >>> fails MOD_UNLOAD events if any locks are outstanding. >> >> Er, no, you just have to acquire the exclusive lock in the MOD_UNLOAD >> handler. >> >> As for the actual availibility of SIX-style locks; I'm fairly sure you >> can do this with the lockmgr. > >Yes. See the allproc_lock as an example. Let's get realistic here. We're not going to get a shared lockmgr lock for every stinking packet that comes into the network. While SIX locks (or semaphore, or shared reader/writer) is nice in theory, I think the performance impact is too much for this particular case. -- Jonathan To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 17:11:42 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 17:11:39 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (Postfix) with ESMTP id CE10C37B401; Mon, 4 Dec 2000 17:11:38 -0800 (PST) Received: (from daemon@localhost) by smtp03.primenet.com (8.9.3/8.9.3) id SAA01652; Mon, 4 Dec 2000 18:09:24 -0700 (MST) Received: from usr02.primenet.com(206.165.6.202) via SMTP by smtp03.primenet.com, id smtpdAAAg7aqid; Mon Dec 4 18:09:13 2000 Received: (from tlambert@localhost) by usr02.primenet.com (8.8.5/8.8.5) id SAA16418; Mon, 4 Dec 2000 18:11:23 -0700 (MST) From: Terry Lambert Message-Id: <200012050111.SAA16418@usr02.primenet.com> Subject: Re: Netgraph and SMP To: jlemon@flugsvamp.com (Jonathan Lemon) Date: Tue, 5 Dec 2000 01:11:23 +0000 (GMT) Cc: jhb@FreeBSD.ORG, smp@FreeBSD.ORG In-Reply-To: <200012042348.eB4NmeA73673@prism.flugsvamp.com> from "Jonathan Lemon" at Dec 04, 2000 05:48:40 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr02.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > >> As for the actual availibility of SIX-style locks; I'm fairly sure you > >> can do this with the lockmgr. > > > >Yes. See the allproc_lock as an example. > > Let's get realistic here. We're not going to get a shared lockmgr > lock for every stinking packet that comes into the network. While > SIX locks (or semaphore, or shared reader/writer) is nice in theory, > I think the performance impact is too much for this particular case. We did 20,000 transactions a second on a P60 with a SIX mode lock manager, back in 1994. Performance is not that big a problem. On the other hand, grabbing it for every packet is not really a great idea. 8-/. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 17:27:31 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 17:27:29 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (Postfix) with ESMTP id 106A437B400 for ; Mon, 4 Dec 2000 17:27:29 -0800 (PST) Received: (from daemon@localhost) by smtp03.primenet.com (8.9.3/8.9.3) id SAA06677; Mon, 4 Dec 2000 18:25:15 -0700 (MST) Received: from usr02.primenet.com(206.165.6.202) via SMTP by smtp03.primenet.com, id smtpdAAAU9ay8m; Mon Dec 4 18:25:07 2000 Received: (from tlambert@localhost) by usr02.primenet.com (8.8.5/8.8.5) id SAA16816; Mon, 4 Dec 2000 18:27:03 -0700 (MST) From: Terry Lambert Message-Id: <200012050127.SAA16816@usr02.primenet.com> Subject: Re: Netgraph and SMP To: cedric@wireless-networks.com (Cedric Berger) Date: Tue, 5 Dec 2000 01:27:02 +0000 (GMT) Cc: smp@FreeBSD.ORG In-Reply-To: <3A2C23ED.14303B3C@wireless-networks.com> from "Cedric Berger" at Dec 04, 2000 03:08:29 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr02.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > How is this SIX-pack lock any different than a standard reader-writer lock? Here's a good introduction to a system that uses this model for its own internal processes: http://www.cs.ndsu.nodak.edu/~tatarino/496/minibase/lockMgr/lockMgr.html The main difference is that multiple reader, single writer locks don't stop new reader locks from being asserted, so long as there is already one or more reader locks already outstanding. This means that if you want a writer lock, you can't have it until all readers are idle. This means that with a lot of readers, a writer can experience starvation deadlock. With intention locks, new reservers aren't allowed to lock until the pending writer has been satisfied. Additional reader requests get in line, so that you don't get similar starvation by prioritizing writers over readers, so if you have tree readers, a writer request, and the three readers drain, the writer is granted. Meanwhile, you get two reader requests and then another writer requests, followed by two more reader requests. You simultaneously grant the reader requests, then they drain, you grant the next writer request, it drains, you grant the next cluster of reader requests. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Mon Dec 4 17:30:20 2000 From owner-freebsd-smp@FreeBSD.ORG Mon Dec 4 17:30:19 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (Postfix) with ESMTP id D316F37B400 for ; Mon, 4 Dec 2000 17:30:18 -0800 (PST) Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id SAA23572; Mon, 4 Dec 2000 18:26:19 -0700 (MST) Received: from usr02.primenet.com(206.165.6.202) via SMTP by smtp04.primenet.com, id smtpdAAAnXaa0T; Mon Dec 4 18:26:08 2000 Received: (from tlambert@localhost) by usr02.primenet.com (8.8.5/8.8.5) id SAA16944; Mon, 4 Dec 2000 18:30:01 -0700 (MST) From: Terry Lambert Message-Id: <200012050130.SAA16944@usr02.primenet.com> Subject: Re: Netgraph and SMP To: cedric@wireless-networks.com (Cedric Berger) Date: Tue, 5 Dec 2000 01:30:01 +0000 (GMT) Cc: smp@FreeBSD.ORG In-Reply-To: <3A2C23ED.14303B3C@wireless-networks.com> from "Cedric Berger" at Dec 04, 2000 03:08:29 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr02.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > How is this SIX-pack lock any different than a standard reader-writer lock? Actually, here is a tutorial from HP, with pictures: http://jazz.external.hp.com/training/sqltables/c5s17.html Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 1:11: 2 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 01:11:00 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mooseriver.com (erie.mooseriver.com [205.166.121.26]) by hub.freebsd.org (Postfix) with ESMTP id ED7BD37B699 for ; Tue, 5 Dec 2000 01:10:57 -0800 (PST) Received: (from jgrosch@localhost) by mooseriver.com (8.11.1/8.9.3) id eB59AtF63413; Tue, 5 Dec 2000 01:10:55 -0800 (PST) (envelope-from jgrosch) Date: Tue, 5 Dec 2000 01:10:55 -0800 From: Josef Grosch To: John Gold Cc: smp@FreeBSD.ORG Subject: Re: dual processor motherboard recommendation for freebsd web/database server Message-ID: <20001205011055.A63208@mooseriver.com> Reply-To: jgrosch@mooseriver.com References: <000201c05d3d$891f6c20$893efea9@kr8> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 1.0.1i In-Reply-To: <000201c05d3d$891f6c20$893efea9@kr8>; from Gold@kr8.com on Sun, Dec 03, 2000 at 03:25:24PM -0000 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Sun, Dec 03, 2000 at 03:25:24PM -0000, John Gold wrote: > Hi, > could anyone recommend a dual processor motherboard for a web/database > server running freebsd? I would like it to support two intel PIII 800 mhz > cpu's (fcpga socket 370) and will be running a raid disk subsystem. So > far the best choice I have found is a Supermicro 370DL3. Does anyone have > any other suggestions or recommendations, good or bad experiences? At work we are using the Intel STL2 with 2 Intel PIII 800 mhz cpu's and 2 gig of ram. These boards can take up to 4 gig with no problem. I am not a big fan of the on-board ether. I have had several (3 or 4) go nuts and spew badly formed IP packets. In these cases we drop in an intel 10/100 either card. YMMV. The on-board video is fine for console access but since we use these as servers in a rack we don't use the video much. Here in San Francisco we are paying $680 for the motherboard. OH! did I mention the on-board SCSI? 2 channels, one wide and the other ultra-wide. Dual IDE channels and 6 PCI slots, 2 64-bit/66MHz and 4 32-bit/33MHz. A damn nice motherboard. If they were just not so damn expensive! Josef -- Josef Grosch | Another day closer to a | FreeBSD 4.2 jgrosch@MooseRiver.com | Micro$oft free world | www.bafug.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 1:41: 5 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 01:41:02 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id AF79937B400; Tue, 5 Dec 2000 01:41:00 -0800 (PST) Received: from timbuktu-59.budapest.interware.hu ([195.70.51.251] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 143Eao-0006zu-00; Tue, 05 Dec 2000 10:40:59 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A2CB814.C0E5841A@elischer.org> Date: Tue, 05 Dec 2000 01:40:36 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: John Baldwin Cc: Mike Smith , smp@FreeBSD.org, phk@FreeBSD.org, archie@FreeBSD.org, brian@FreeBSD.org, Brian Somers Subject: Re: Netgraph and SMP References: Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org John Baldwin wrote: > > On 04-Dec-00 Mike Smith wrote: > >> > The simplest structure for this is a shared/exclusive lock > >> > that supports intention; Terry would have ranted about this. (He would > >> > have called it a SIX-lock, I think). > >> [.....] > >> > This may sound simplistic, but given that you don't necessarily make > >> > changes to Netgraph very often, this is quite likely more than adequate > >> > for now. > >> > >> Nice, I never realised there were shared/exclusive locks available. > >> I think netgraph nodes would also need to have a ``modevent'' that > >> fails MOD_UNLOAD events if any locks are outstanding. > > > > Er, no, you just have to acquire the exclusive lock in the MOD_UNLOAD > > handler. > > > > As for the actual availibility of SIX-style locks; I'm fairly sure you > > can do this with the lockmgr. > > Yes. See the allproc_lock as an example. uh, folks.. the lock manager is a bloated pig.. if you think I'm going to call lock-manager for every packet as it passes through every node in its path, well, I think we have easier ways of slowing down the system..... > > -- > > John Baldwin -- http://www.FreeBSD.org/~jhb/ > PGP Key: http://www.baldwin.cx/~john/pgpkey.asc > "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 2:36:55 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 02:36:53 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (Postfix) with ESMTP id 040BE37B400; Tue, 5 Dec 2000 02:36:53 -0800 (PST) Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id DAA28488; Tue, 5 Dec 2000 03:32:23 -0700 (MST) Received: from usr05.primenet.com(206.165.6.205) via SMTP by smtp04.primenet.com, id smtpdAAAkTaOH3; Tue Dec 5 03:32:15 2000 Received: (from tlambert@localhost) by usr05.primenet.com (8.8.5/8.8.5) id DAA27764; Tue, 5 Dec 2000 03:36:07 -0700 (MST) From: Terry Lambert Message-Id: <200012051036.DAA27764@usr05.primenet.com> Subject: Re: Netgraph and SMP To: julian@elischer.org (Julian Elischer) Date: Tue, 5 Dec 2000 10:36:06 +0000 (GMT) Cc: jhb@FreeBSD.ORG (John Baldwin), msmith@FreeBSD.ORG (Mike Smith), smp@FreeBSD.ORG, phk@FreeBSD.ORG, archie@FreeBSD.ORG, brian@FreeBSD.ORG, brian@Awfulhak.org (Brian Somers) In-Reply-To: <3A2CB814.C0E5841A@elischer.org> from "Julian Elischer" at Dec 05, 2000 01:40:36 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr05.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > uh, folks.. the lock manager is a bloated pig.. > if you think I'm going to call lock-manager for every packet as it passes > through every node in its path, well, I think we have easier ways of slowing > down the system..... Agreed. The lock manager should not be used to implement either semaphores or mutexes. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 5:19:25 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 05:19:22 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id B4FF437B699; Tue, 5 Dec 2000 05:19:15 -0800 (PST) Received: from luanda-20.budapest.interware.hu ([195.70.51.20] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 143I02-0005GJ-00; Tue, 05 Dec 2000 14:19:14 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A2CE99B.CBD857CD@elischer.org> Date: Tue, 05 Dec 2000 05:11:55 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: Mike Smith Cc: brian@freebsd.org, archie@freebsd.org, phk@freebsd.org, smp@freebsd.org Subject: Re: Netgraph and SMP References: <200012042140.eB4Le9F01294@mass.osd.bsdi.com> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Mike Smith wrote: > > The simplest structure for this is a shared/exclusive lock > that supports intention; Terry would have ranted about this. (He would > have called it a SIX-lock, I think). > > In this model, you acquire the lock in 'shared' mode every time you enter > Netgraph, and release it when you leave. > > When you plan to make changes to Netgraph, you get the lock in > 'exclusive' mode. 'Intention' comes in here; now that you are trying to > get the lock in exclusive mode, your intention is recorded and nobody > else can get it in 'shared' mode, so eventually all the users drain out > of Netgraph and you get the lock. > > This may sound simplistic, but given that you don't necessarily make > changes to Netgraph very often, this is quite likely more than adequate > for now. I'd agree with this as long as getting and releasing READ mode locks is very fast in the standard case. You are right. Setup times are really quite irrelevent unless of course you want to have completely uninterrupted flow in one part of the graph while you change something elsewhere in the graph. But I think even that would be acceptable if coded right.. One possibility I was thinking about was to break the overall system into 'graphs'. A very busy heavily networked system might have several disjoint graphs. There would be no point in locking a graph you were not going to alter. I'm going to check out the locks available to see what I can use. > > > I'm trying to figure out the locking needed for SMP in Netgraph. > > > > Here are some of the various constraints that I see. (some of theme are not just > > SMP issues). > > > > 1/ can't remove modules that are being used (the base framework) or have living > > nodes (other netgraph modules). > > 2/ Cannot remove a node if it is in use, (or in a return stack somewhere) > > 3/ Cannot unlink an 'edge' if someone is passing data across it. > > 4/ cannot use edges and nodes that are in the process of being set up or torn > > down. > > 5/ Defered deliveries cannot be made to nodes that are being removed. > > 6/ Nodes cannot be completely removed if there are pending deliveries for them. > > > > What locking schemes to otther people use for protecting intricate structures of > > objects with various interconnections? The buffer cache is one example that > > comes to mind, as is the routing table. > > > > Julian > > -- > > __--_|\ Julian Elischer > > / \ julian@elischer.org > > ( OZ ) World tour 2000 > > ---> X_.---._/ presently in: Budapest > > v > > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > > with "unsubscribe freebsd-smp" in the body of the message > > > > -- > ... every activity meets with opposition, everyone who acts has his > rivals and unfortunately opponents also. But not because people want > to be opponents, rather because the tasks and relationships force > people to take different points of view. [Dr. Fritz Todt] > V I C T O R Y N O T V E N G E A N C E -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 8:23: 9 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 08:23:08 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (unknown [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 1828937B400; Tue, 5 Dec 2000 08:23:07 -0800 (PST) Received: from berserker.bsdi.com (cp@localhost.bsdi.com [127.0.0.1]) by berserker.bsdi.com (8.11.1/8.9.3) with ESMTP id eB5GMtm00562; Tue, 5 Dec 2000 09:22:55 -0700 (MST) (envelope-from cp@berserker.bsdi.com) Message-Id: <200012051622.eB5GMtm00562@berserker.bsdi.com> To: Julian Elischer Cc: Mike Smith , brian@FreeBSD.org, archie@FreeBSD.org, phk@FreeBSD.org, smp@FreeBSD.org Subject: Re: Netgraph and SMP In-reply-to: Your message of "Mon, 04 Dec 2000 13:40:08 PST." <200012042140.eB4Le9F01294@mass.osd.bsdi.com> From: Chuck Paterson Date: Tue, 05 Dec 2000 09:22:55 -0700 Sender: cp@berserker.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Julian, A standard lock manager lock will provide the functionality Mike talks about below. Chuck Mike Smith wrote on: Mon, 04 Dec 2000 13:40:08 PST } }The simplest structure for this is a shared/exclusive lock }that supports intention; Terry would have ranted about this. (He would }have called it a SIX-lock, I think). } }In this model, you acquire the lock in 'shared' mode every time you enter }Netgraph, and release it when you leave. } }When you plan to make changes to Netgraph, you get the lock in }'exclusive' mode. 'Intention' comes in here; now that you are trying to }get the lock in exclusive mode, your intention is recorded and nobody }else can get it in 'shared' mode, so eventually all the users drain out }of Netgraph and you get the lock. } }This may sound simplistic, but given that you don't necessarily make }changes to Netgraph very often, this is quite likely more than adequate }for now. } To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 8:31:44 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 08:31:42 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from crewsoft.com (ns.aenet.net [157.22.214.1]) by hub.freebsd.org (Postfix) with ESMTP id 2D60B37B400 for ; Tue, 5 Dec 2000 08:31:42 -0800 (PST) Received: from [63.197.8.222] (HELO wireless-networks.com) by crewsoft.com (CommuniGate Pro SMTP 3.3b5) with ESMTP id 369624; Tue, 05 Dec 2000 08:33:59 -0800 Message-ID: <3A2D18A1.48C89595@wireless-networks.com> Date: Tue, 05 Dec 2000 08:32:33 -0800 From: Cedric Berger X-Mailer: Mozilla 4.73 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Terry Lambert Cc: smp@FreeBSD.ORG Subject: Six-Lock? (was Re: Netgraph and SMP) References: <200012050130.SAA16944@usr02.primenet.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Ok, Thanks for the pointer. I guess I (and many people) use use the term reader/writer lock for exactly the "SIX-lock" mechanism you describes. In fact, I always thought that whether or not readers stop entering the lock when a writer request it is a matter of "lock policy". (You can even imagine any mixed policy where new readers are allowed to enter when a write is pending to improve throughoutput, but only for during a limited time to avoid writer starvation) Cedric Terry Lambert wrote: > > > How is this SIX-pack lock any different than a standard reader-writer lock? > > Actually, here is a tutorial from HP, with pictures: > > http://jazz.external.hp.com/training/sqltables/c5s17.html > > Terry Lambert > terry@lambert.org > --- > Any opinions in this posting are my own and not those of my present > or previous employers. > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 8:52:57 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 08:52:55 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 0F5A537B400; Tue, 5 Dec 2000 08:52:55 -0800 (PST) Received: from berserker.bsdi.com (cp@localhost.bsdi.com [127.0.0.1]) by berserker.bsdi.com (8.11.1/8.9.3) with ESMTP id eB5Gqgm02018; Tue, 5 Dec 2000 09:52:42 -0700 (MST) (envelope-from cp@berserker.bsdi.com) Message-Id: <200012051652.eB5Gqgm02018@berserker.bsdi.com> To: Jonathan Lemon Cc: jhb@FreeBSD.org, smp@FreeBSD.org Subject: Re: Netgraph and SMP In-reply-to: Your message of "Mon, 04 Dec 2000 17:48:40 CST." <200012042348.eB4NmeA73673@prism.flugsvamp.com> From: Chuck Paterson Date: Tue, 05 Dec 2000 09:52:42 -0700 Sender: cp@berserker.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org The vast majority of the cost of the lock manager locks is the mutex release and acquire. On Intel at least it is possible to have reader/write locks that are half as expensive in the non-contested cased. HOWEVER, we don't have these primitive now and reader writer locks with the lock manager are as cheap as we have got. We probably out to wrap the acquire and release in macros so we can swap them out later. Right now we just really need to get the functionality. I would also like to point out that the cost of all the atomic operations being put in the path collectively make the lock manager lock just not that big of a deal. This is not something I like, but just something to put things in perspective. Chuck } }Let's get realistic here. We're not going to get a shared lockmgr }lock for every stinking packet that comes into the network. While }SIX locks (or semaphore, or shared reader/writer) is nice in theory, }I think the performance impact is too much for this particular case. }-- }Jonathan } } }To Unsubscribe: send mail to majordomo@FreeBSD.org }with "unsubscribe freebsd-smp" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 10:44:52 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 10:44:50 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id 4091C37B400 for ; Tue, 5 Dec 2000 10:44:49 -0800 (PST) Received: from casablanca-41.budapest.interware.hu ([195.70.53.41] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 143N54-0004Ob-00 for ; Tue, 05 Dec 2000 19:44:46 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A2D3786.5E8CC39F@elischer.org> Date: Tue, 05 Dec 2000 10:44:22 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: smp@freebsd.org Subject: Mutex types. Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In my quest to try get netgraph SMP compatible, I've been diving into the mutex code. Several things are noticiable. 1/ These aren't really very light weight. There are cases when what is wanted is a spin-mudex that takes a couple of instructions at most and is held for maybe 4 instructions. The mutexes implemented in kern/kern_mutex.c are very heavyweight, and not very suitable for fine-grain work. is there any plan to introduce lighter contructs? From my perspective I KNOW I won't want to recurse so I don't want all that recursion gumpf. 2/ There is no such thing as a reader/writer (SIX?) mutex in the set. Are there any plans for such? Of course one could be implemted usinghte existing mutexes, but as they are so bloated anyhow, an attempt to do so would produce something in the 'godzilla' range. 3/ The attempt to combine all types of mutexes into one has various side effects. One being the sentence in the man page: ---- It is an error to acquire a mutex in one mode (e.g. spin) and release it in another (e.g. default). It is also an error to get the lock in one mode and al- low another thread to attempt to get the lock in another mode. A good general rule is to always use a given mutex in one mode only. ---- If they were differnt structures, then it wouldn't be POSSIBLE to get them wrong if typechecking were used in the compiler. I cannot believe that the attempt to make one 'all-singing-and-dancing' mutex doesn't make the code and data bigger. What were the reasons for it to be done this way? 4/ I notice the comment in the man page: ----- This could result in deadlock if a thread interrupted the thread which held a mutex and then tried to acquire the mutex; for this reason spin locks will disable all interrupts (on the local CPU on- ly) by default. ----- How expensive is it to disable all interrupts? Summary. I have a preference to having different types of mutexes with seperate specialised functions for them. And a (SIX?) mutex would definitly be one of them, as would a NON RECURSIVE spinlock. (that spins for about 100 machine cycles before panicing when in debug mode). Overloading them as we are now is I think a suboptimal move.. -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 10:45: 6 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 10:45:00 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id E297337B402; Tue, 5 Dec 2000 10:44:57 -0800 (PST) Received: from casablanca-41.budapest.interware.hu ([195.70.53.41] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 143N5D-0004Pm-00; Tue, 05 Dec 2000 19:44:55 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A2D03A6.B36803A2@elischer.org> Date: Tue, 05 Dec 2000 07:03:02 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: Brian Somers Cc: smp@freebsd.org, archie@freebsd.org Subject: Re: Netgraph and SMP References: <200012042307.eB4N7gD93751@hak.lan.Awfulhak.org> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Brian Somers wrote: > > Er, no, you just have to acquire the exclusive lock in the MOD_UNLOAD > > handler. > > Is it desirable to lock up running kldunload(8) ? Personally I think it would be easier if the unload request were simply refused for any module that is "in use". Unfortunatly there is no real standard for this yet as far as I can see. many modules just refuse to unload to solve the problem, and some just unload blindly, hoping that there are no users. If I protect netgraph with a single user/modifier lock, as Mike suggests (not a bad idea that) so that users can hold off a modifier, I still face problems at the boundaries. If the lock is in the Netgraph base code, which is a module, I can have the problem where I have to ask the code that is being removed, if it has any locks, yet I am in a race with another process doing the same. If The other process is a bit ahead, then just my attempt to gain the lock itself will result in a segfault as I access the place where the lock USED to be. We have that sort of problem all over the place. For example: We can have races between processes accessing /dev entries racing with other processes removing the driver itself. There is no lock on the devsw table that I am aware of (yet). Particularly one that every device open needs to hold. It seems to me that such a VERY FAST but specialised user/modifier locking primitive would come in useful all over the place. Unloading modules present a real nightmare scenario, because you can't even trust a function call into the module to tell you if it's available. That function call or any locks that are in the module may just 'go away' one instruction before you use them. In a worst case scenario you should hold some special lock every time you cross a boundary into code that is supplied by a different module than one you already have some hold over (whether that hold comes in the form of a lock or a reference count, or an 'open count' doesn't matter as long as it stops the module from being unloaded. What happens if someone tries to do a 'mount -t dosfs' just as someone else decides to unload the dosfs module? We have no strategy that I am aware of to deal with this sort of thing. For now I have to make do with hoping that a device that is turned off will not generate traffic, and that hopefully, driver modules that have running instances will refuse to unload. -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 11: 3:20 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 11:03:17 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from Awfulhak.org (tun.AwfulHak.org [194.242.139.173]) by hub.freebsd.org (Postfix) with ESMTP id DD0E937B400; Tue, 5 Dec 2000 11:02:57 -0800 (PST) Received: from hak.lan.Awfulhak.org (root@hak.lan.awfulhak.org [172.16.0.12]) by Awfulhak.org (8.11.1/8.11.1) with ESMTP id eB5IxBM08486; Tue, 5 Dec 2000 18:59:11 GMT (envelope-from brian@hak.lan.Awfulhak.org) Received: from hak.lan.Awfulhak.org (brian@localhost [127.0.0.1]) by hak.lan.Awfulhak.org (8.11.1/8.11.1) with ESMTP id eB5J20Y38879; Tue, 5 Dec 2000 19:02:00 GMT (envelope-from brian@hak.lan.Awfulhak.org) Message-Id: <200012051902.eB5J20Y38879@hak.lan.Awfulhak.org> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: Julian Elischer Cc: Brian Somers , smp@FreeBSD.org, archie@FreeBSD.org, brian@Awfulhak.org Subject: Re: Netgraph and SMP In-Reply-To: Message from Julian Elischer of "Tue, 05 Dec 2000 07:03:02 PST." <3A2D03A6.B36803A2@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 05 Dec 2000 19:02:00 +0000 From: Brian Somers Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I don't think netgraph is the place to deal with this. kldunload should be smart enough to do several things: Look at the dependency graph that it maintains (does it?) and decide if there are any dependent modules. If they are, refuse the unload request. Inside it's own lock (preventing any other dependent modules from appearing), it asks the module if it's ok to unload. This is obviously more tricky than it sounds. We've got to ensure that if any character device entry points have been created (make_dev() etc), we block the relevant entry points so that we can ENODEV if the MOD_UNLOAD works. Apart from the generic cdevsw entry points and module dependencies, I think it's pretty much up to the module to ensure that things work - unless someone can think of another way that an external source can use the module (sysctls spring to mind). > Brian Somers wrote: > > > > Er, no, you just have to acquire the exclusive lock in the MOD_UNLOAD > > > handler. > > > > Is it desirable to lock up running kldunload(8) ? > > Personally I think it would be easier if the unload request were > simply refused for any module that is "in use". > > Unfortunatly there is no real standard for this yet as far > as I can see. many modules just refuse to unload to solve > the problem, and some just unload blindly, hoping that > there are no users. > > If I protect netgraph with a single user/modifier lock, as > Mike suggests (not a bad idea that) so that users > can hold off a modifier, I still face problems at the boundaries. > > If the lock is in the Netgraph base code, which is a module, > I can have the problem where I have to ask the code that is being > removed, if it has any locks, yet I am in a race with another > process doing the same. If The other process is a bit ahead, then > just my attempt to gain the lock itself will result in a segfault > as I access the place where the lock USED to be. > > We have that sort of problem all over the place. > > For example: > We can have races between processes accessing /dev entries > racing with other processes removing the driver itself. > There is no lock on the devsw table that I am aware of (yet). > Particularly one that every device open needs to hold. > It seems to me that such a VERY FAST but specialised > user/modifier locking primitive would come in useful > all over the place. Unloading modules present a real > nightmare scenario, because you can't even trust a function > call into the module to tell you if it's available. That > function call or any locks that are in the module may > just 'go away' one instruction before you use them. > In a worst case scenario you should hold some special > lock every time you cross a boundary into code that is > supplied by a different module than one you already have > some hold over (whether that hold comes in the form of a > lock or a reference count, or an 'open count' doesn't > matter as long as it stops the module from being unloaded. > > What happens if someone tries to do a 'mount -t dosfs' > just as someone else decides to unload the dosfs module? > > We have no strategy that I am aware of to deal with this > sort of thing. > > For now I have to make do with hoping that a device that > is turned off will not generate traffic, and that hopefully, > driver modules that have running instances will refuse to unload. > > -- > __--_|\ Julian Elischer > / \ julian@elischer.org > ( OZ ) World tour 2000 > ---> X_.---._/ presently in: Budapest > v -- Brian Don't _EVER_ lose your sense of humour ! To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 12:45: 2 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 12:45:00 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (c228380-a.sfmissn1.sfba.home.com [24.20.90.44]) by hub.freebsd.org (Postfix) with ESMTP id CA10037B400 for ; Tue, 5 Dec 2000 12:44:59 -0800 (PST) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.11.0/8.11.1) with ESMTP id eB5Ks3F01085; Tue, 5 Dec 2000 12:54:03 -0800 (PST) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200012052054.eB5Ks3F01085@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Julian Elischer Cc: smp@freebsd.org Subject: Re: Mutex types. In-reply-to: Your message of "Tue, 05 Dec 2000 10:44:22 PST." <3A2D3786.5E8CC39F@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Tue, 05 Dec 2000 12:54:03 -0800 From: Mike Smith Sender: msmith@mass.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > In my quest to try get netgraph SMP compatible, I've been diving into the > mutex code. Several things are noticiable. I think you need to spend some time reading over the earlier discussions on this material, as you've got some items out of perspective here, particularly the cost of some of the operations you're considering using. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 13: 8: 6 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 13:08:03 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from magnesium.net (toxic.magnesium.net [207.154.84.15]) by hub.freebsd.org (Postfix) with SMTP id 5EEE237B400 for ; Tue, 5 Dec 2000 13:08:03 -0800 (PST) Received: (qmail 38096 invoked by uid 1142); 5 Dec 2000 21:08:02 -0000 Date: 5 Dec 2000 13:08:02 -0800 Date: Tue, 5 Dec 2000 13:07:55 -0800 From: Jason Evans To: Julian Elischer Cc: smp@freebsd.org Subject: Re: Mutex types. Message-ID: <20001205130755.C2312@canonware.com> References: <3A2D3786.5E8CC39F@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3A2D3786.5E8CC39F@elischer.org>; from julian@elischer.org on Tue, Dec 05, 2000 at 10:44:22AM -0800 Sender: jasone@magnesium.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, Dec 05, 2000 at 10:44:22AM -0800, Julian Elischer wrote: > 1/ These aren't really very light weight. > There are cases when what is wanted is a spin-mudex that takes a couple of > instructions at most and is held for maybe 4 instructions. The mutexes > implemented in kern/kern_mutex.c > are very heavyweight, and not very suitable for fine-grain work. If you look closely at the code that actually gets executed in the common case, the mutex code is pretty streamlined. In the common case, the code in kern/kern_mutex.c never even gets called; the real action is in sys/mutex.h. Getting rid of recursion (actually making recursion available via a separate synchronization primitive) would slightly improve performance, but it isn't huge difference. > 2/ There is no such thing as a reader/writer (SIX?) mutex in the set. > Are there any plans for such? Of course one could be implemted usinghte existing > mutexes, but as they are so bloated anyhow, an attempt to do so would produce > something in the 'godzilla' range. Building SIX locks on top of mutexes would IMO be the correct way to go. Really, the mutex code isn't so bad if you look closely at what it does. > 3/ The attempt to combine all types of mutexes into one has various side > effects. > One being the sentence in the man page: > ---- > It is an error > to acquire a mutex in one mode (e.g. spin) and release it in another > (e.g. default). It is also an error to get the lock in one mode and al- > low another thread to attempt to get the lock in another mode. A good > general rule is to always use a given mutex in one mode only. > ---- > If they were differnt structures, then it wouldn't be POSSIBLE to get them wrong > if typechecking > were used in the compiler. I cannot believe that the attempt to make one > 'all-singing-and-dancing' mutex doesn't make the code and data bigger. > What were the reasons for it to be done this way? We inherited this interface from the BSD/OS code. I don't think anyone completely likes the API (in particular, having to pass in MTX_DEF or MTX_SPIN with every operation), but up to now there have been higher priorities than changing the mutex API. The main reason for the original design was flexibility during conversion of the kernel. That said, I would like to see the recursive and spin mutex support split out. Even more though, I'd like to see major portions of the kernel running outside of Giant. > 4/ I notice the comment in the man page: > ----- > This could result in deadlock if a thread interrupted > the thread which held a mutex and then tried to acquire the mutex; for > this reason spin locks will disable all interrupts (on the local CPU on- > ly) by default. > ----- > How expensive is it to disable all interrupts? I dunno, but correctness takes precedence over performance in my book. =) Jason To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 14: 2:59 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 14:02:57 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id AE93737B400 for ; Tue, 5 Dec 2000 14:02:56 -0800 (PST) Received: from berserker.bsdi.com (cp@localhost.bsdi.com [127.0.0.1]) by berserker.bsdi.com (8.11.1/8.9.3) with ESMTP id eB5M2mm04439; Tue, 5 Dec 2000 15:02:48 -0700 (MST) (envelope-from cp@berserker.bsdi.com) Message-Id: <200012052202.eB5M2mm04439@berserker.bsdi.com> To: Jason Evans Cc: Julian Elischer , smp@FreeBSD.ORG Subject: Re: Mutex types. In-reply-to: Your message of "Tue, 05 Dec 2000 13:07:55 PST." <20001205130755.C2312@canonware.com> From: Chuck Paterson Date: Tue, 05 Dec 2000 15:02:48 -0700 Sender: cp@berserker.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org }Building SIX locks on top of mutexes would IMO be the correct way to go. BSD/OS doesn't have low level reader/write locks because the lock manager had the basic functionality and we haven't got to the point were chasing the performance has come close to the top of the queue. Having said this, there is a case for actually building low level reader/writer locks. They can be made to as fast in the uncontested case as a mutex, at least on hardware like Sparc, Intel and Alpha. Reader/Write Locks built on top of mutices will be roughly half as fast as a mutex because a mutex acquire and release is needed for both acquiring and releasing the reader/write locks. For this reason I believe we should just stick with lock manager locks until we have a chance to build real reader write locks. There are tricks which can be done by mixing atomic counting and swap and compare to gate entry. I pretty clearly understand how this can be done, but its still unclear to me how well this maps onto standard counting semaphores. If the protection for the netgraph is encapsulated in netgraph specific macros we ought to be able to switch this out to whatever mechanism we choose when we have time to design/implement one or more of the alternatives. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 15:21:10 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 15:21:08 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from monk.via.net (monk.via.net [209.81.2.10]) by hub.freebsd.org (Postfix) with ESMTP id 4BBD337B401 for ; Tue, 5 Dec 2000 15:21:08 -0800 (PST) Received: (from joe@localhost) by monk.via.net (8.11.0/8.11.0) id eB5NMM480841 for smp@FreeBSD.ORG; Tue, 5 Dec 2000 15:22:22 -0800 (PST) (envelope-from joe) From: Joe McGuckin Message-Id: <200012052322.eB5NMM480841@monk.via.net> Date: Tue, 5 Dec 2000 15:21:52 -0800 (PST) To: smp@FreeBSD.ORG Subject: Re[2]: BSD/OS interrupt code In-Reply-To: X-Mailer: Ishmail 1.3.1-970608-bsdi MIME-Version: 1.0 Content-Type: text/plain Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hey, I've been out of town for a week & just saw this... Doesn't this sound similar to Henry Massalin's and Calton Pu's synthesis kernel papers? Where they dynamically generate specifically compiled O/S entry points? Instead of having one read() system call (for example) with various state and constraint checks in the code, a custom read() entry point would be generated. It was extremely fast. As part of this work, they also created a superoptimizer that would produce very compact and fast code sequences for use in the runtime code generator. My guess is that they were pre-generating function stub routines & performing limited instruction substitution in the stub routines to customize it. Supposedly the superoptimizer reduced the doprnt() routine (it used to be the guts of printf(), etc) to something like 800 bytes, but it took weeks to generate the code! joe -- Joe McGuckin ViaNet Communications 994 San Antonio Road Palo Alto, CA 94303 Phone: 650-969-2203 Cell: 650-207-0372 Fax: 650-969-2124 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 22:56: 4 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 22:56:02 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mobile.wemm.org (adsl-64-163-195-99.dsl.snfc21.pacbell.net [64.163.195.99]) by hub.freebsd.org (Postfix) with ESMTP id 290F837B400 for ; Tue, 5 Dec 2000 22:56:02 -0800 (PST) Received: from netplex.com.au (localhost [127.0.0.1]) by mobile.wemm.org (8.11.1/8.11.1) with ESMTP id eB66twJ29960 for ; Tue, 5 Dec 2000 22:56:01 -0800 (PST) (envelope-from peter@netplex.com.au) Message-Id: <200012060656.eB66twJ29960@mobile.wemm.org> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: smp@freebsd.org Subject: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fwd) Date: Tue, 05 Dec 2000 22:55:58 -0800 From: Peter Wemm Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org FYI: I have seen a few folks with compaq DL360's post on the list.. I'd appreciate a few more testers. This should backport to RELENG_4 trivially. http://people.freebsd.org/~peter/compaq.diff Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au "All of this is for nothing if we don't go to the stars" - JMS/B5 ------- Forwarded Message peter 2000/12/05 19:47:14 PST Modified files: sys/i386/i386 mp_machdep.c Log: This is kind of a nasty hack, but it appears to solve the Compaq DL360 SMP problem. Compaq, in their infinite wisdom, forgot to put the IO apic intpin #0 connection to the 8259 PIC into the mptable. This hack is to look and see if intpin #0 has *no* table entry and adds a fake ExtInt entry for the remap routines to use. isa/clock.c will still test the interrupts. This entry is only ever used on an already broken system. Revision Changes Path 1.131 +14 -3 src/sys/i386/i386/mp_machdep.c ------- End of Forwarded Message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Tue Dec 5 22:56:18 2000 From owner-freebsd-smp@FreeBSD.ORG Tue Dec 5 22:56:16 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by hub.freebsd.org (Postfix) with ESMTP id 9AD6437B401 for ; Tue, 5 Dec 2000 22:56:14 -0800 (PST) Received: from bde.zeta.org.au (bde.zeta.org.au [203.2.228.102]) by mailman.zeta.org.au (8.8.7/8.8.7) with ESMTP id RAA15095; Wed, 6 Dec 2000 17:56:01 +1100 Date: Wed, 6 Dec 2000 17:56:13 +1100 (EST) From: Bruce Evans X-Sender: bde@besplex.bde.org To: Julian Elischer Cc: smp@FreeBSD.ORG Subject: Re: Mutex types. In-Reply-To: <3A2D3786.5E8CC39F@elischer.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Tue, 5 Dec 2000, Julian Elischer wrote: > In my quest to try get netgraph SMP compatible, I've been diving into the > mutex code. Several things are noticiable. > > 1/ These aren't really very light weight. > There are cases when what is wanted is a spin-mudex that takes a couple of > instructions at most and is held for maybe 4 instructions. The mutexes > implemented in kern/kern_mutex.c > are very heavyweight, and not very suitable for fine-grain work. is there any > plan to introduce lighter contructs? From my perspective I KNOW I won't want to > recurse so I don't want all that recursion gumpf. The mutexes implemented in kern/kern_mutex.c are rarely called. Most of the implementation is in and , and most of the code there is normally optimized away after compile-time flags tests show that most of it is dead. That said, mutexes aren't _really_ _very_ light weight. The compile-time flags tests require inline functions or macros to work, and the inline code isn't very small. GENERIC grew by 100K text (5%) when (null, !SMP) simple locks were changed to mutexes. > 3/ The attempt to combine all types of mutexes into one has various side > effects. I dislike this too. > ... > were used in the compiler. I cannot believe that the attempt to make one > 'all-singing-and-dancing' mutex doesn't make the code and data bigger. The compile-time flags tests eliminate most of the code bloat (except without optimization of course). > 4/ I notice the comment in the man page: > ----- > This could result in deadlock if a thread interrupted > the thread which held a mutex and then tried to acquire the mutex; for > this reason spin locks will disable all interrupts (on the local CPU on- > ly) by default. > ----- > How expensive is it to disable all interrupts? Very expensive for latency. It makes interrupt latency on a 486DX2/66 worse than it used to be on a 386SX/16. (The 486DX2/66 was the slowest machine I could find to test on :-). It was just slow enough to demonstrate breakage of fast interrupt handling for sio.) I have unfinished fixes for this. (Essentially, upgrade to the old interrupt masking scheme again -- use the core of its internals for the same reasons as before, although not the driver interface (spl*())). This only works right on machines that have per-interrupt interrupt masking, but so does our general interrupt scheme.) Bruce To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 3: 5: 7 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 03:05:06 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from averroe.polito.it (averroe.polito.it [130.192.4.32]) by hub.freebsd.org (Postfix) with SMTP id 7FDF937B400 for ; Wed, 6 Dec 2000 03:05:05 -0800 (PST) Received: (qmail 92990 invoked by uid 10003); 6 Dec 2000 11:04:56 -0000 Received: from demostene.polito.it (HELO demostene.sb.polito.it) (130.192.4.73) by averroe.polito.it with SMTP; 6 Dec 2000 11:04:56 -0000 Message-Id: <5.0.1.4.0.20001206104633.00a800f0@sbpop.polito.it> X-Sender: tealdi@sbpop.polito.it X-Mailer: QUALCOMM Windows Eudora Version 5.0.1 Date: Wed, 06 Dec 2000 12:04:57 +0100 To: Peter Wemm , smp@freebsd.org From: paolo tealdi Subject: Re: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fwd) In-Reply-To: <200012060656.eB66twJ29960@mobile.wemm.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org At 22.55 05/12/00 -0800, Peter Wemm wrote: >FYI: I have seen a few folks with compaq DL360's post on the list.. I'd >appreciate a few more testers. This should backport to RELENG_4 trivially. I'm very (very very) happy to tell you all that applying this patch, the kernel seems to boot (at least), the server sees the two processors and all seem to be ok. In any case if i see instabilities, i'll tell you . Thanks for the patch. Best regards, Paolo Tealdi Ing. Paolo Tealdi Library & News System Administrator Politecnico Torino Phone : +39-011-5646714 , FAX : +39-011-5646799 C.so D. degli Abruzzi, 24 - 10129 Torino - ITALY Email : tealdi@sb.polito.it To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 3:49:38 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 03:49:36 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mobile.wemm.org (adsl-64-163-195-99.dsl.snfc21.pacbell.net [64.163.195.99]) by hub.freebsd.org (Postfix) with ESMTP id 511B837B400 for ; Wed, 6 Dec 2000 03:49:36 -0800 (PST) Received: from netplex.com.au (localhost [127.0.0.1]) by mobile.wemm.org (8.11.1/8.11.1) with ESMTP id eB6BnT602822; Wed, 6 Dec 2000 03:49:29 -0800 (PST) (envelope-from peter@netplex.com.au) Message-Id: <200012061149.eB6BnT602822@mobile.wemm.org> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: paolo tealdi Cc: smp@FreeBSD.ORG Subject: Re: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fwd) In-Reply-To: <5.0.1.4.0.20001206104633.00a800f0@sbpop.polito.it> Date: Wed, 06 Dec 2000 03:49:29 -0800 From: Peter Wemm Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org paolo tealdi wrote: > At 22.55 05/12/00 -0800, Peter Wemm wrote: > >FYI: I have seen a few folks with compaq DL360's post on the list.. I'd > >appreciate a few more testers. This should backport to RELENG_4 trivially. > > > I'm very (very very) happy to tell you all that applying this patch, the > kernel seems to boot (at least), the server sees the two processors and all > seem to be ok. > In any case if i see instabilities, i'll tell you . I assume this was under RELENG_4? If so, thanks for the confirmation. As far as stability goes, all this patch does is tweak the interrupt routing descriptor tables so that we can boot. After that, it has no effect on things. Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 9:48: 4 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 09:48:01 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from thelab.hub.org (CDR22-166.accesscable.net [24.138.22.166]) by hub.freebsd.org (Postfix) with ESMTP id 1768137B400 for ; Wed, 6 Dec 2000 09:47:56 -0800 (PST) Received: from localhost (scrappy@localhost) by thelab.hub.org (8.11.1/8.11.1) with ESMTP id eB6HlGm17320 for ; Wed, 6 Dec 2000 13:47:16 -0400 (AST) (envelope-from scrappy@hub.org) X-Authentication-Warning: thelab.hub.org: scrappy owned process doing -bs Date: Wed, 6 Dec 2000 13:47:16 -0400 (AST) From: The Hermit Hacker To: freebsd-smp@freebsd.org Subject: IBM 7100 Server with Dual Xeon's ... Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Morning all ... I've been trying to get FreeBSD to go SMP on this box for ages now ... since *BSD isn't supported by IBM, I got one of our linux guys over here to help me out ... we created an SMP boot floppy, to see if we'd get a hang using it, and we didn't *but*, it did give us a warning: WARNING: MP table in the EBDA can be UNSAFE does this mean anything to anyone? As part of the Linux boot sequence, it also told us: Intel MultiProcessor Specifications v1.4 VirtualWire Compatibility mode. if that also means anything? Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 11:30:40 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 11:30:38 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-176-64.dsl.snfc21.pacbell.net [63.202.176.64]) by hub.freebsd.org (Postfix) with ESMTP id 3F1BE37B400 for ; Wed, 6 Dec 2000 11:30:38 -0800 (PST) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.11.0/8.11.1) with ESMTP id eB6JdOF37209; Wed, 6 Dec 2000 11:39:36 -0800 (PST) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200012061939.eB6JdOF37209@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Bruce Evans Cc: smp@FreeBSD.ORG Subject: Interrupt latency (was Re: Mutex types. ) In-reply-to: Your message of "Wed, 06 Dec 2000 17:56:13 +1100." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 06 Dec 2000 11:39:24 -0800 From: Mike Smith Sender: msmith@mass.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > ... This > only works right on machines that have per-interrupt interrupt masking, > but so does our general interrupt scheme.) Since most of the machines we're likely to run on for the next few years *don't* have this feature, I'd say it would make a lot more sense to spend some effort on devising an interrupt scheme that worked well without that assumption... -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 12:27:20 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 12:27:18 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-176-64.dsl.snfc21.pacbell.net [63.202.176.64]) by hub.freebsd.org (Postfix) with ESMTP id A666837B400 for ; Wed, 6 Dec 2000 12:27:17 -0800 (PST) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.11.0/8.11.1) with ESMTP id eB6KaDF52151; Wed, 6 Dec 2000 12:36:19 -0800 (PST) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200012062036.eB6KaDF52151@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: The Hermit Hacker Cc: freebsd-smp@freebsd.org Subject: Re: IBM 7100 Server with Dual Xeon's ... In-reply-to: Your message of "Wed, 06 Dec 2000 13:47:16 -0400." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Wed, 06 Dec 2000 12:36:13 -0800 From: Mike Smith Sender: msmith@mass.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Try the patch that Peter just posted for the Compaq DL360. I think your box has the same problem. > I've been trying to get FreeBSD to go SMP on this box for ages now > ... since *BSD isn't supported by IBM, I got one of our linux guys over > here to help me out ... we created an SMP boot floppy, to see if we'd get > a hang using it, and we didn't *but*, it did give us a warning: > > WARNING: MP table in the EBDA can be UNSAFE > > does this mean anything to anyone? > > As part of the Linux boot sequence, it also told us: > > Intel MultiProcessor Specifications v1.4 > VirtualWire Compatibility mode. > > if that also means anything? > > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy > Systems Administrator @ hub.org > primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 13:47:40 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 13:47:38 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from thelab.hub.org (CDR22-166.accesscable.net [24.138.22.166]) by hub.freebsd.org (Postfix) with ESMTP id AFBB137B401; Wed, 6 Dec 2000 13:47:36 -0800 (PST) Received: from localhost (scrappy@localhost) by thelab.hub.org (8.11.1/8.11.1) with ESMTP id eB6Lkx720114; Wed, 6 Dec 2000 17:46:59 -0400 (AST) (envelope-from scrappy@hub.org) X-Authentication-Warning: thelab.hub.org: scrappy owned process doing -bs Date: Wed, 6 Dec 2000 17:46:59 -0400 (AST) From: The Hermit Hacker To: Mike Smith Cc: freebsd-smp@freebsd.org Subject: Re: IBM 7100 Server with Dual Xeon's ... In-Reply-To: <200012062036.eB6KaDF52151@mass.osd.bsdi.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org damn, I was *going* to ask ... can someone resend it, and does it apply to 4.2? thanks ... On Wed, 6 Dec 2000, Mike Smith wrote: > > Try the patch that Peter just posted for the Compaq DL360. I think your > box has the same problem. > > > I've been trying to get FreeBSD to go SMP on this box for ages now > > ... since *BSD isn't supported by IBM, I got one of our linux guys over > > here to help me out ... we created an SMP boot floppy, to see if we'd get > > a hang using it, and we didn't *but*, it did give us a warning: > > > > WARNING: MP table in the EBDA can be UNSAFE > > > > does this mean anything to anyone? > > > > As part of the Linux boot sequence, it also told us: > > > > Intel MultiProcessor Specifications v1.4 > > VirtualWire Compatibility mode. > > > > if that also means anything? > > > > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy > > Systems Administrator @ hub.org > > primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org > > > > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > > with "unsubscribe freebsd-smp" in the body of the message > > > > -- > ... every activity meets with opposition, everyone who acts has his > rivals and unfortunately opponents also. But not because people want > to be opponents, rather because the tasks and relationships force > people to take different points of view. [Dr. Fritz Todt] > V I C T O R Y N O T V E N G E A N C E > > > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 14: 4:51 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 14:04:49 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from elvis.mu.org (elvis.mu.org [207.154.226.10]) by hub.freebsd.org (Postfix) with ESMTP id E44D437B400; Wed, 6 Dec 2000 14:04:48 -0800 (PST) Received: by elvis.mu.org (Postfix, from userid 1000) id 7B3972B2DB; Wed, 6 Dec 2000 16:04:43 -0600 (CST) Date: Wed, 6 Dec 2000 14:04:43 -0800 From: Paul Saab To: The Hermit Hacker Cc: Mike Smith , freebsd-smp@freebsd.org Subject: Re: IBM 7100 Server with Dual Xeon's ... Message-ID: <20001206140443.A38809@elvis.mu.org> References: <200012062036.eB6KaDF52151@mass.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: ; from scrappy@hub.org on Wed, Dec 06, 2000 at 05:46:59PM -0400 Sender: ps@mu.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Yes it applied to 4.x http://people.freebsd.org/~peter/compaq.diff The Hermit Hacker (scrappy@hub.org) wrote: > > damn, I was *going* to ask ... can someone resend it, and does it apply to > 4.2? > > thanks ... > > On Wed, 6 Dec 2000, Mike Smith wrote: > > > > > Try the patch that Peter just posted for the Compaq DL360. I think your > > box has the same problem. > > > > > I've been trying to get FreeBSD to go SMP on this box for ages now > > > ... since *BSD isn't supported by IBM, I got one of our linux guys over > > > here to help me out ... we created an SMP boot floppy, to see if we'd get > > > a hang using it, and we didn't *but*, it did give us a warning: > > > > > > WARNING: MP table in the EBDA can be UNSAFE > > > > > > does this mean anything to anyone? > > > > > > As part of the Linux boot sequence, it also told us: > > > > > > Intel MultiProcessor Specifications v1.4 > > > VirtualWire Compatibility mode. > > > > > > if that also means anything? > > > > > > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy > > > Systems Administrator @ hub.org > > > primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org > > > > > > > > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > > > with "unsubscribe freebsd-smp" in the body of the message > > > > > > > -- > > ... every activity meets with opposition, everyone who acts has his > > rivals and unfortunately opponents also. But not because people want > > to be opponents, rather because the tasks and relationships force > > people to take different points of view. [Dr. Fritz Todt] > > V I C T O R Y N O T V E N G E A N C E > > > > > > > > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy > Systems Administrator @ hub.org > primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message -- Paul Saab Technical Yahoo paul@mu.org - ps@yahoo-inc.com - ps@freebsd.org Do You .. uhh .. Yahoo!? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 16: 4:37 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 16:04:35 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id 166DA37B400; Wed, 6 Dec 2000 16:04:34 -0800 (PST) Received: from kampala-40.budapest.interware.hu ([195.70.52.232] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 143oXy-0007qv-00; Thu, 07 Dec 2000 01:04:26 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A2E520E.5E4444BD@elischer.org> Date: Wed, 06 Dec 2000 06:49:50 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: Brian Somers Cc: smp@FreeBSD.org, archie@FreeBSD.org Subject: Re: Netgraph and SMP References: <200012051902.eB5J20Y38879@hak.lan.Awfulhak.org> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Brian Somers wrote: > > I don't think netgraph is the place to deal with this. > > kldunload should be smart enough to do several things: > > Look at the dependency graph that it maintains (does it?) and > decide if there are any dependent modules. If they are, refuse > the unload request. I be;ieve this is already done (but am not totally sure) > > Inside it's own lock (preventing any other dependent modules from > appearing), it asks the module if it's ok to unload. > > This is obviously more tricky than it sounds. We've got to ensure > that if any character device entry points have been created > (make_dev() etc), we block the relevant entry points so that we can > ENODEV if the MOD_UNLOAD works. but what if the process has already entered the driver? > > Apart from the generic cdevsw entry points and module dependencies, I > think it's pretty much up to the module to ensure that things work - > unless someone can think of another way that an external source can > use the module (sysctls spring to mind). yep > -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 18: 0:42 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 18:00:40 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from Awfulhak.org (awfulhak.demon.co.uk [194.222.196.252]) by hub.freebsd.org (Postfix) with ESMTP id 3997037B400; Wed, 6 Dec 2000 18:00:38 -0800 (PST) Received: from hak.lan.Awfulhak.org (root@hak.lan.awfulhak.org [172.16.0.12]) by Awfulhak.org (8.11.1/8.11.1) with ESMTP id eB71wpx19082; Thu, 7 Dec 2000 01:58:51 GMT (envelope-from brian@lan.awfulhak.org) Received: from hak.lan.Awfulhak.org (brian@localhost [127.0.0.1]) by hak.lan.Awfulhak.org (8.11.1/8.11.1) with ESMTP id eB721Yt44116; Thu, 7 Dec 2000 02:01:34 GMT (envelope-from brian@hak.lan.Awfulhak.org) Message-Id: <200012070201.eB721Yt44116@hak.lan.Awfulhak.org> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: Julian Elischer Cc: Brian Somers , smp@FreeBSD.org, archie@FreeBSD.org, brian@Awfulhak.org Subject: Re: Netgraph and SMP In-Reply-To: Message from Julian Elischer of "Wed, 06 Dec 2000 06:49:50 PST." <3A2E520E.5E4444BD@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 07 Dec 2000 02:01:33 +0000 From: Brian Somers Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > > Inside it's own lock (preventing any other dependent modules from > > appearing), it asks the module if it's ok to unload. > > > > This is obviously more tricky than it sounds. We've got to ensure > > that if any character device entry points have been created > > (make_dev() etc), we block the relevant entry points so that we can > > ENODEV if the MOD_UNLOAD works. > > but what if the process has already entered the driver? Then it should hold a lock that fails in the kldload syscall when it does a mutex_trylock() on it. Again, this is somewhere that a read/ write mutex would be handy. kldunload ends up trying for the write lock while driver entry points try for the read lock. -- Brian Don't _EVER_ lose your sense of humour ! To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Wed Dec 6 23:12: 5 2000 From owner-freebsd-smp@FreeBSD.ORG Wed Dec 6 23:12:03 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp.OpenBit.NET (smtp.OpenBit.NET [211.14.152.40]) by hub.freebsd.org (Postfix) with ESMTP id 81C3C37B400 for ; Wed, 6 Dec 2000 23:12:02 -0800 (PST) Received: from hermite.FreeBit.NET (dhcp1003.FreeBit.NET [211.14.152.251]) by smtp.OpenBit.NET (8.10.2/8.10.2) with SMTP id eB7791314324; Thu, 7 Dec 2000 16:09:04 +0900 (JST) Message-Id: <200012070711.AA02140@hermite.FreeBit.NET> From: IWAIZAKO Takahiro Date: Thu, 07 Dec 2000 16:11:41 +0900 To: Peter Wemm Cc: smp@FreeBSD.ORG Subject: Re: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fwd) In-Reply-To: <200012060656.eB66twJ29960@mobile.wemm.org> References: <200012060656.eB66twJ29960@mobile.wemm.org> MIME-Version: 1.0 X-Mailer: AL-Mail32 Version 1.11 Content-Type: text/plain; charset=us-ascii Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Peter Wemm wrote: >FYI: I have seen a few folks with compaq DL360's post on the list.. I'd >appreciate a few more testers. This should backport to RELENG_4 trivially. > >http://people.freebsd.org/~peter/compaq.diff I applied this patch to 4.2-STABLE on DL360, it may work fine. Many many thanks! avail memory = 649240576 (634024K bytes) APIC_IO: MP table broken: 8259->APIC entry missing! Changing APIC ID for IO APIC #0 from 0 to 8 on chip Programming 35 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 3, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 8, version: 0x00220011, at 0xfec00000 Preloaded elf kernel "kernel" at 0xc0359000. Preloaded userconfig_script "/boot/kernel.conf" at 0xc035909c. Pentium Pro MTRR support enabled ... snip APIC_IO: Testing 8254 interrupt delivery APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin 2 APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 SMP: AP CPU #1 Launched! --- rabit@FreeBit.NET To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 10:52:13 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 10:52:10 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id D473D37B400; Thu, 7 Dec 2000 10:52:08 -0800 (PST) Received: from luanda-33.budapest.interware.hu ([195.70.51.33] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 144697-0004J0-00; Thu, 07 Dec 2000 19:51:57 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A2F98CA.B3FD1B40@elischer.org> Date: Thu, 07 Dec 2000 06:03:54 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: Brian Somers Cc: smp@FreeBSD.org, archie@FreeBSD.org Subject: Re: Netgraph and SMP References: <200012070201.eB721Yt44116@hak.lan.Awfulhak.org> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Brian Somers wrote: > > > > > > > Inside it's own lock (preventing any other dependent modules from > > > appearing), it asks the module if it's ok to unload. > > > > > > This is obviously more tricky than it sounds. We've got to ensure > > > that if any character device entry points have been created > > > (make_dev() etc), we block the relevant entry points so that we can > > > ENODEV if the MOD_UNLOAD works. > > > > but what if the process has already entered the driver? > > Then it should hold a lock that fails in the kldload syscall when it > does a mutex_trylock() on it. Again, this is somewhere that a read/ > write mutex would be handy. kldunload ends up trying for the write > lock while driver entry points try for the read lock. what happens in the gap between 'looking up the driver' and getting the lock for the driver? What if the driver is unloaded in that gap. This means that every operation via the devsw table needs to gain a spinlock or something to keep them apart. > > -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 10:52:16 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 10:52:08 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id B144A37B402 for ; Thu, 7 Dec 2000 10:52:07 -0800 (PST) Received: from luanda-33.budapest.interware.hu ([195.70.51.33] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 14469E-0004K1-00; Thu, 07 Dec 2000 19:52:04 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A2FDC0A.9F628C4B@elischer.org> Date: Thu, 07 Dec 2000 10:50:50 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: Chuck Paterson Cc: Jason Evans , smp@FreeBSD.ORG, Archie@packetdesign.com Subject: Re: Mutex types. References: <200012052202.eB5M2mm04439@berserker.bsdi.com> Content-Type: text/plain; charset=iso-8859-15 Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Chuck Paterson wrote: > > }Building SIX locks on top of mutexes would IMO be the correct way to go. > > BSD/OS doesn't have low level reader/write locks because > the lock manager had the basic functionality and we haven't got to > the point were chasing the performance has come close to the top > of the queue. Having said this, there is a case for actually building > low level reader/writer locks. They can be made to as fast in the > uncontested case as a mutex, at least on hardware like Sparc, > Intel and Alpha. Reader/Write Locks built on top of mutices will > be roughly half as fast as a mutex because a mutex acquire and > release is needed for both acquiring and releasing the reader/write > locks. For this reason I believe we should just stick with lock > manager locks until we have a chance to build real reader write > locks. I am tempted to write some simple reader-writer locks (low level) just for the fun of it.. > > There are tricks which can be done by mixing atomic counting > and swap and compare to gate entry. I pretty clearly understand > how this can be done, but its still unclear to me how well this > maps onto standard counting semaphores. > > If the protection for the netgraph is encapsulated in netgraph > specific macros we ought to be able to switch this out to whatever > mechanism we choose when we have time to design/implement one or more > of the alternatives. > > Chuck Since netgraph users are NOT generaly allowed to sleep we could make it a SPIN type lock. It would be up to the writing callers (probably netgraph Macros) to use these to quickly do their damage, and release asap. I think that we could almost get away with one oe two of these per netgraph system. (As mike smith suggested). (around the boundaries for packets in the system, and some for critical datastructures) Just for fun.. Here's what the unoptimised pseudocode might look like. Active writer sets the LSB in the writers count. each waiting writer increments it by 2. Having looked at the present mutex stuff, I think I know how these would assemble and they probably wouldn't be too slow to be run on a 'per packet' basis. These are designed to be used by a higher level framework that knows what to do when the request fails. It might sleep the thread or it might do something else and retry.... Struct foo { tid_t mtx; int readers; int writers; } /* try get read permission.. only fails if there writers running or waiting */ request_read_try(struct foo *mtx) { int retval push_and_disable_ints(); while (atomic_test_and_set(&mtx->mtx, MTX_UNOWNED, MTX_OWNED) == FAIL) { /* DO SOMETHING THAT LEAVES THE BUS FREE FOR A FEW CYCLES */ } if ((retval = mtx->writers) == 0) { mtx->readers++; } mtx->mtx = MTX_UNOWNED; pop_ints(); return(retval); /* cif !0, caller should check for a waiting writer */ } /* try get permission to write. Say if we already reserved a table */ request_write_try(struct foo *mtx, int first) { push_and_disable_ints(); while (atomic_test_and_set(&mtx->mtx, MTX_UNOWNED, MTX_OWNED) == FAIL) { /* DO SOMETHING THAT LEAVES THE BUS FREE FOR A FEW CYCLES */ } if ((mtx->readers == 0) && ((mtx_writers & 1) == 0) { if (!first) /* -2 + 1 == -1 */ mtx->writers--; /* no longer waiting */ else mtx->writers++; mtx->mtx = MTX_UNOWNED; restore_ints(stored_flags); return (SUCCEEDED) } else { if (first) mtx->writers += 2; mtx->mtx = MTX_UNOWNED; pop_ints(); return(FAILED); } release_read(struct foo mtx) { push_and_disable_ints(); while (atomic_test_and_set(&mtx->mtx, MTX_UNOWNED, MTX_OWNED) == FAIL) { /* DO SOMETHING THAT LEAVES THE BUS FREE FOR A FEW CYCLES */ } mtx->readers--; mtx->mtx = MTX_UNOWNED; pop_ints(); return (mtx_readers) /* if 0 caller should check for sleeping writers */ } release_write(struct foo mtx) { push_and_disable_ints(); while (atomic_test_and_set(&mtx->mtx, MTX_UNOWNED, MTX_OWNED) == FAIL) { /* DO SOMETHING THAT LEAVES THE BUS FREE FOR A FEW CYCLES */ } mtx->writers--; mtx->mtx = MTX_UNOWNED; pop_ints(); } /* caller should always check if readers or writers sleeping */ /* Give up re-trying for while, remove our reservation, */ release_pending_write(struct foo mtx) { push_and_disable_ints(); while (atomic_test_and_set(&mtx->mtx, MTX_UNOWNED, MTX_OWNED) == FAIL) { /* DO SOMETHING THAT LEAVES THE BUS FREE FOR A FEW CYCLES */ } mtx->writers -= 2; mtx->mtx = MTX_UNOWNED; pop_ints(); } -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 12:22: 3 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 12:22:00 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from thelab.hub.org (CDR22-166.accesscable.net [24.138.22.166]) by hub.freebsd.org (Postfix) with ESMTP id 8477937B400 for ; Thu, 7 Dec 2000 12:21:59 -0800 (PST) Received: from localhost (scrappy@localhost) by thelab.hub.org (8.11.1/8.11.1) with ESMTP id eB7KKS835872; Thu, 7 Dec 2000 16:20:28 -0400 (AST) (envelope-from scrappy@hub.org) X-Authentication-Warning: thelab.hub.org: scrappy owned process doing -bs Date: Thu, 7 Dec 2000 16:20:28 -0400 (AST) From: The Hermit Hacker To: IWAIZAKO Takahiro Cc: Peter Wemm , smp@FreeBSD.ORG Subject: Re: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fwd) In-Reply-To: <200012070711.AA02140@hermite.FreeBit.NET> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org fixes me too on the IBM 7100 Quad: APIC_IO: Testing 8254 interrupt delivery APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin 2 APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 now, since both myself and IWAIZAKO are getting the exact same error, is it actually broken? :) this is Compaq vs IBM ... Any chance of getting this patch into the 4.x tree? :) On Thu, 7 Dec 2000, IWAIZAKO Takahiro wrote: > Peter Wemm wrote: > >FYI: I have seen a few folks with compaq DL360's post on the list.. I'd > >appreciate a few more testers. This should backport to RELENG_4 trivially. > > > >http://people.freebsd.org/~peter/compaq.diff > > > I applied this patch to 4.2-STABLE on DL360, it may work fine. > Many many thanks! > > avail memory = 649240576 (634024K bytes) > APIC_IO: MP table broken: 8259->APIC entry missing! > Changing APIC ID for IO APIC #0 from 0 to 8 on chip > Programming 35 pins in IOAPIC #0 > IOAPIC #0 intpin 2 -> irq 0 > FreeBSD/SMP: Multiprocessor motherboard > cpu0 (BSP): apic id: 3, version: 0x00040011, at 0xfee00000 > cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee00000 > io0 (APIC): apic id: 8, version: 0x00220011, at 0xfec00000 > Preloaded elf kernel "kernel" at 0xc0359000. > Preloaded userconfig_script "/boot/kernel.conf" at 0xc035909c. > Pentium Pro MTRR support enabled > > ... snip > > APIC_IO: Testing 8254 interrupt delivery > APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin 2 > APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 > SMP: AP CPU #1 Launched! > > --- > rabit@FreeBit.NET > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 12:27:21 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 12:27:19 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 8EEB137B400 for ; Thu, 7 Dec 2000 12:27:19 -0800 (PST) Received: from foo.osd.bsdi.com (root@foo.osd.bsdi.com [204.216.28.137]) by pike.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id eB7KQd726479; Thu, 7 Dec 2000 12:26:39 -0800 (PST) (envelope-from jhb@foo.osd.bsdi.com) Received: (from jhb@localhost) by foo.osd.bsdi.com (8.11.1/8.11.0) id eB7KQg465394; Thu, 7 Dec 2000 12:26:42 -0800 (PST) (envelope-from jhb) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Thu, 07 Dec 2000 12:26:42 -0800 (PST) Organization: BSD, Inc. From: John Baldwin To: The Hermit Hacker Subject: Re: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fw Cc: smp@FreeBSD.ORG, Peter Wemm , IWAIZAKO Takahiro Sender: jhb@foo.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 07-Dec-00 The Hermit Hacker wrote: > > fixes me too on the IBM 7100 Quad: > > APIC_IO: Testing 8254 interrupt delivery > APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpin > 2 > APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 > > now, since both myself and IWAIZAKO are getting the exact same error, is > it actually broken? :) this is Compaq vs IBM ... They are both broken. :) This definitely is a workaround for some broken motherboards out there, not a case of us not handling a set of non-broken motherboards. :) > Any chance of getting this patch into the 4.x tree? :) Peter? -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.Baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 13:14:34 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 13:14:32 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from net2.gendyn.com (nat2.gendyn.com [204.60.171.12]) by hub.freebsd.org (Postfix) with ESMTP id 7971E37B400 for ; Thu, 7 Dec 2000 13:14:32 -0800 (PST) Received: from [153.11.11.3] (helo=plunger.gdeb.com) by net2.gendyn.com with esmtp (Exim 2.12 #1) id 1448Mq-0002ui-00 for smp@freebsd.org; Thu, 7 Dec 2000 16:14:16 -0500 Received: from orion.caen.gdeb.com ([153.11.109.11]) by plunger.gdeb.com with ESMTP id QAA14547 for ; Thu, 7 Dec 2000 16:10:05 -0500 (EST) Received: from vigrid.com (gpz.clc.gdeb.com [192.168.3.12]) by orion.caen.gdeb.com (8.9.3/8.9.3) with ESMTP id QAA16440 for ; Thu, 7 Dec 2000 16:10:26 -0500 (EST) (envelope-from eischen@vigrid.com) Sender: eghk@gdeb.com Message-ID: <3A2FFDD9.F17C1165@vigrid.com> Date: Thu, 07 Dec 2000 16:15:05 -0500 From: Dan Eischen X-Mailer: Mozilla 4.75 [en] (X11; U; SunOS 5.8 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: smp@freebsd.org Subject: Userland atomic assignments Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org What kind of atomic operations can we get in userland? If I have a singly linked list, where nodes are only added to the head or the tail and they are never removed, can I walk the list without fear of catching a bad head or node->next (when I'm at the end of the list) pointer? In other words, can I get an atomic_set_ptr() operation on each platform? I can also see the need for an atomic_set_int32() operation. Thanks, -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 14: 7:54 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 14:07:52 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from magnesium.net (toxic.magnesium.net [207.154.84.15]) by hub.freebsd.org (Postfix) with SMTP id 6E3E237B401 for ; Thu, 7 Dec 2000 14:07:52 -0800 (PST) Received: (qmail 23405 invoked by uid 1142); 7 Dec 2000 22:07:51 -0000 Date: 7 Dec 2000 14:07:51 -0800 Date: Thu, 7 Dec 2000 14:07:46 -0800 From: Jason Evans To: Dan Eischen Cc: smp@freebsd.org Subject: Re: Userland atomic assignments Message-ID: <20001207140746.O2312@canonware.com> References: <3A2FFDD9.F17C1165@vigrid.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <3A2FFDD9.F17C1165@vigrid.com>; from eischen@vigrid.com on Thu, Dec 07, 2000 at 04:15:05PM -0500 Sender: jasone@magnesium.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thu, Dec 07, 2000 at 04:15:05PM -0500, Dan Eischen wrote: > What kind of atomic operations can we get in userland? > If I have a singly linked list, where nodes are only added > to the head or the tail and they are never removed, can I > walk the list without fear of catching a bad head or > node->next (when I'm at the end of the list) pointer? > > In other words, can I get an atomic_set_ptr() operation > on each platform? I can also see the need for an > atomic_set_int32() operation. I don't think we have an official API for atomic operations in userland. There is , but it isn't documented as a userland API. I agree that we need this or something similar in userland though. Any ideas on the cleanest way to provide this? Jason To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 14: 9:30 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 14:09:29 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 2B29137B400 for ; Thu, 7 Dec 2000 14:09:29 -0800 (PST) Received: from laptop.baldwin.cx (root@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id eB7M8s730300; Thu, 7 Dec 2000 14:08:54 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: <3A2FFDD9.F17C1165@vigrid.com> Date: Thu, 07 Dec 2000 14:09:43 -0800 (PST) From: John Baldwin To: Dan Eischen Subject: RE: Userland atomic assignments Cc: smp@FreeBSD.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 07-Dec-00 Dan Eischen wrote: > What kind of atomic operations can we get in userland? > If I have a singly linked list, where nodes are only added > to the head or the tail and they are never removed, can I > walk the list without fear of catching a bad head or > node->next (when I'm at the end of the list) pointer? > > In other words, can I get an atomic_set_ptr() operation > on each platform? I can also see the need for an > atomic_set_int32() operation. atomic_cmpset() can't be used because it dinks with interrupts in the i386 case. Also, you would need to #define SMP so that the lock prefixes would be compiled in for the x86 case. You are probably better off just using userland mutexes. Also, not that atomic_set() does an or operation, whereas atomic_store() does an actual assignment. > Thanks, > > -- > Dan Eischen -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 14:20:21 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 14:20:19 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 9CBC537B401 for ; Thu, 7 Dec 2000 14:20:18 -0800 (PST) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id RAA11017; Thu, 7 Dec 2000 17:19:53 -0500 (EST) Date: Thu, 7 Dec 2000 17:19:53 -0500 (EST) From: Daniel Eischen To: Jason Evans Cc: Dan Eischen , smp@freebsd.org Subject: Re: Userland atomic assignments In-Reply-To: <20001207140746.O2312@canonware.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 7 Dec 2000, Jason Evans wrote: > On Thu, Dec 07, 2000 at 04:15:05PM -0500, Dan Eischen wrote: > > What kind of atomic operations can we get in userland? > > If I have a singly linked list, where nodes are only added > > to the head or the tail and they are never removed, can I > > walk the list without fear of catching a bad head or > > node->next (when I'm at the end of the list) pointer? > > > > In other words, can I get an atomic_set_ptr() operation > > on each platform? I can also see the need for an > > atomic_set_int32() operation. > > I don't think we have an official API for atomic operations in userland. > There is , but it isn't documented as a userland API. I > agree that we need this or something similar in userland though. Any ideas > on the cleanest way to provide this? Something along the lines of I'd guess. I don't need compare and set, nor anything that complicated, just a couple of atomic store operations. I'm finding some places in libc that could really use an atomic_store_ptr(). -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 14:23: 5 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 14:23:04 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 7D66D37B400; Thu, 7 Dec 2000 14:23:03 -0800 (PST) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id RAA11512; Thu, 7 Dec 2000 17:22:42 -0500 (EST) Date: Thu, 7 Dec 2000 17:22:42 -0500 (EST) From: Daniel Eischen To: John Baldwin Cc: Dan Eischen , smp@FreeBSD.org Subject: RE: Userland atomic assignments In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thu, 7 Dec 2000, John Baldwin wrote: > atomic_cmpset() can't be used because it dinks with interrupts in the i386 > case. Also, you would need to #define SMP so that the lock prefixes would be > compiled in for the x86 case. You are probably better off just using userland > mutexes. Also, not that atomic_set() does an or operation, whereas > atomic_store() does an actual assignment. I don't need cmpset operations, just atomic int32 and ptr assignments. The latter would be good enough to let me walk a simple list without having to take a lock. -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 14:26: 4 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 14:26:02 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pike.osd.bsdi.com (pike.osd.bsdi.com [204.216.28.222]) by hub.freebsd.org (Postfix) with ESMTP id 8D09337B400 for ; Thu, 7 Dec 2000 14:26:02 -0800 (PST) Received: from laptop.baldwin.cx (root@jhb-laptop.osd.bsdi.com [204.216.28.241]) by pike.osd.bsdi.com (8.11.1/8.9.3) with ESMTP id eB7MPQ730849; Thu, 7 Dec 2000 14:25:26 -0800 (PST) (envelope-from jhb@FreeBSD.org) Message-ID: X-Mailer: XFMail 1.4.0 on FreeBSD X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit MIME-Version: 1.0 In-Reply-To: Date: Thu, 07 Dec 2000 14:26:15 -0800 (PST) From: John Baldwin To: Daniel Eischen Subject: Re: Userland atomic assignments Cc: smp@FreeBSD.org, Jason Evans Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 07-Dec-00 Daniel Eischen wrote: > On 7 Dec 2000, Jason Evans wrote: >> On Thu, Dec 07, 2000 at 04:15:05PM -0500, Dan Eischen wrote: >> > What kind of atomic operations can we get in userland? >> > If I have a singly linked list, where nodes are only added >> > to the head or the tail and they are never removed, can I >> > walk the list without fear of catching a bad head or >> > node->next (when I'm at the end of the list) pointer? >> > >> > In other words, can I get an atomic_set_ptr() operation >> > on each platform? I can also see the need for an >> > atomic_set_int32() operation. >> >> I don't think we have an official API for atomic operations in userland. >> There is , but it isn't documented as a userland API. I >> agree that we need this or something similar in userland though. Any ideas >> on the cleanest way to provide this? > > Something along the lines of I'd guess. > I don't need compare and set, nor anything that complicated, > just a couple of atomic store operations. I'm finding some > places in libc that could really use an atomic_store_ptr(). a = b; should be atomic I think. If its not, then large portions of the kernel break, IIRC. > -- > Dan Eischen -- John Baldwin -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 14:33:36 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 14:33:34 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from pcnet1.pcnet.com (pcnet1.pcnet.com [204.213.232.3]) by hub.freebsd.org (Postfix) with ESMTP id 9B00C37B400; Thu, 7 Dec 2000 14:33:33 -0800 (PST) Received: (from eischen@localhost) by pcnet1.pcnet.com (8.8.7/PCNet) id RAA12984; Thu, 7 Dec 2000 17:33:12 -0500 (EST) Date: Thu, 7 Dec 2000 17:33:11 -0500 (EST) From: Daniel Eischen To: John Baldwin Cc: smp@FreeBSD.org, Jason Evans Subject: Re: Userland atomic assignments In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Thu, 7 Dec 2000, John Baldwin wrote: > On 07-Dec-00 Daniel Eischen wrote: > > Something along the lines of I'd guess. > > I don't need compare and set, nor anything that complicated, > > just a couple of atomic store operations. I'm finding some > > places in libc that could really use an atomic_store_ptr(). > > a = b; should be atomic I think. If its not, then large portions of the kernel > break, IIRC. OK, thanks. I just didn't want to make any assumptions about archs I didn't know. -- Dan Eischen To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 18:13:40 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 18:13:37 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 4870B37B400; Thu, 7 Dec 2000 18:13:34 -0800 (PST) Received: (from grog@localhost) by wantadilla.lemis.com (8.11.0/8.9.3) id eB82DUX37145; Fri, 8 Dec 2000 12:43:30 +1030 (CST) (envelope-from grog) Date: Fri, 8 Dec 2000 12:43:30 +1030 From: Greg Lehey To: FreeBSD current users Cc: FreeBSD SMP list Subject: Hangs during 'make world -j4' on recent current Message-ID: <20001208124330.Z27667@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: grog@wantadilla.lemis.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org For over a week now, I have been unable to complete a 'make world' on my -CURRENT box if I specify -j4. The system hangs and is completely unresponsive. This is a dual Celeron and an Abit BP6 motherboard. As far as I can tell, nobody knows what's causing this, nor even how to attack the problem. I'd like to solicit feedback about the extent of the problem, the possible causes, and how to debug it. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 19:11:20 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 19:11:18 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp05.primenet.com (smtp05.primenet.com [206.165.6.135]) by hub.freebsd.org (Postfix) with ESMTP id 7B55437B400; Thu, 7 Dec 2000 19:11:18 -0800 (PST) Received: (from daemon@localhost) by smtp05.primenet.com (8.9.3/8.9.3) id UAA14568; Thu, 7 Dec 2000 20:07:52 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp05.primenet.com, id smtpdAAAiKaGBC; Thu Dec 7 20:07:45 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id UAA02374; Thu, 7 Dec 2000 20:11:08 -0700 (MST) From: Terry Lambert Message-Id: <200012080311.UAA02374@usr08.primenet.com> Subject: Re: Netgraph and SMP To: brian@Awfulhak.org (Brian Somers) Date: Fri, 8 Dec 2000 03:11:08 +0000 (GMT) Cc: julian@elischer.org (Julian Elischer), brian@Awfulhak.org (Brian Somers), smp@FreeBSD.ORG, archie@FreeBSD.ORG In-Reply-To: <200012070201.eB721Yt44116@hak.lan.Awfulhak.org> from "Brian Somers" at Dec 07, 2000 02:01:33 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr08.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > Inside it's own lock (preventing any other dependent modules from > > > appearing), it asks the module if it's ok to unload. > > > > > > This is obviously more tricky than it sounds. We've got to ensure > > > that if any character device entry points have been created > > > (make_dev() etc), we block the relevant entry points so that we can > > > ENODEV if the MOD_UNLOAD works. > > > > but what if the process has already entered the driver? > > Then it should hold a lock that fails in the kldload syscall when it > does a mutex_trylock() on it. Again, this is somewhere that a read/ > write mutex would be handy. kldunload ends up trying for the write > lock while driver entry points try for the read lock. In Solaris, the entry into the driver would hold a reference, which would result in the reference count being incremented. Only modules with a 0 reference count can be unloaded. This same mechanism is used for vnodes, and for modules on which other modules depend. It works well, ans is very light weight. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 19:23:41 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 19:23:39 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-176-64.dsl.snfc21.pacbell.net [63.202.176.64]) by hub.freebsd.org (Postfix) with ESMTP id D7F7C37B400 for ; Thu, 7 Dec 2000 19:23:38 -0800 (PST) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.11.0/8.11.1) with ESMTP id eB83WtF00456; Thu, 7 Dec 2000 19:32:55 -0800 (PST) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200012080332.eB83WtF00456@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Terry Lambert Cc: smp@FreeBSD.ORG Subject: Re: Netgraph and SMP In-reply-to: Your message of "Fri, 08 Dec 2000 03:11:08 GMT." <200012080311.UAA02374@usr08.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 07 Dec 2000 19:32:55 -0800 From: Mike Smith Sender: msmith@mass.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > In Solaris, the entry into the driver would hold a reference, > which would result in the reference count being incremented. > Only modules with a 0 reference count can be unloaded. This > same mechanism is used for vnodes, and for modules on which > other modules depend. It works well, ans is very light weight. The whole problem is that it *isn't* very light weight. The reference count has to be atomic, which means that it ping-pongs around from CPU to CPU, causing a lot of extra cache traffic. OTOH, there's not much we can do about this short of going looking for better multi-CPU reference count implementations once we have time to worry about performance. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 19:50:18 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 19:50:16 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (Postfix) with ESMTP id 994E537B400; Thu, 7 Dec 2000 19:50:15 -0800 (PST) Received: (from daemon@localhost) by smtp02.primenet.com (8.9.3/8.9.3) id UAA22273; Thu, 7 Dec 2000 20:45:38 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp02.primenet.com, id smtpdAAApOaWDR; Thu Dec 7 20:45:30 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id UAA03298; Thu, 7 Dec 2000 20:50:04 -0700 (MST) From: Terry Lambert Message-Id: <200012080350.UAA03298@usr08.primenet.com> Subject: Re: Netgraph and SMP To: msmith@FreeBSD.ORG (Mike Smith) Date: Fri, 8 Dec 2000 03:50:04 +0000 (GMT) Cc: tlambert@primenet.com (Terry Lambert), smp@FreeBSD.ORG In-Reply-To: <200012080332.eB83WtF00456@mass.osd.bsdi.com> from "Mike Smith" at Dec 07, 2000 07:32:55 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr08.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > In Solaris, the entry into the driver would hold a reference, > > which would result in the reference count being incremented. > > Only modules with a 0 reference count can be unloaded. This > > same mechanism is used for vnodes, and for modules on which > > other modules depend. It works well, ans is very light weight. > > The whole problem is that it *isn't* very light weight. > > The reference count has to be atomic, which means that it ping-pongs > around from CPU to CPU, causing a lot of extra cache traffic. > > OTOH, there's not much we can do about this short of going looking for > better multi-CPU reference count implementations once we have time to > worry about performance. Actually, you can just put it in non-cacheable memory, and the penalty will only be paid by the CPU(s) doing the referencing. This means a clock multiplier worth of cycles, though, to get it in and out of main memory from the CPU. Back when all this started, clock multipliers weren't 1/5th the problem they pose today... Still, for a very large number of CPUs, this would work fine for all but frequently contended objects. I think that it is making more and more sense to lock interrupts to a single CPU. What happens if you write to a page that's marked non-cachable on the CPU on which you are running, but cacheable on another CPU? Does it do the right thing, and update the cache on the caching CPU? If so, locking the interrupt processing for each card to a particular CPU could be very worthwhile, since you would never take the hit, unless you were doing something extraordinary. BTW, you would want to grab a ref, check for the heavy lock, and back off it it were held. The unload would want to grab the heavy lock, grab a ref, and do its work when everyone is backed off (ref = 1). This ordering would ensure the least overhead for the normal case (no heavy lock, ref > 0). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 21:24:11 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 21:24:09 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-176-64.dsl.snfc21.pacbell.net [63.202.176.64]) by hub.freebsd.org (Postfix) with ESMTP id F202537B400 for ; Thu, 7 Dec 2000 21:24:07 -0800 (PST) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.11.0/8.11.1) with ESMTP id eB85XRN00458; Thu, 7 Dec 2000 21:33:27 -0800 (PST) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200012080533.eB85XRN00458@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: Terry Lambert Cc: smp@FreeBSD.ORG Subject: Re: Netgraph and SMP In-reply-to: Your message of "Fri, 08 Dec 2000 03:50:04 GMT." <200012080350.UAA03298@usr08.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 07 Dec 2000 21:33:27 -0800 From: Mike Smith Sender: msmith@mass.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > > In Solaris, the entry into the driver would hold a reference, > > > which would result in the reference count being incremented. > > > Only modules with a 0 reference count can be unloaded. This > > > same mechanism is used for vnodes, and for modules on which > > > other modules depend. It works well, ans is very light weight. > > > > The whole problem is that it *isn't* very light weight. > > > > The reference count has to be atomic, which means that it ping-pongs > > around from CPU to CPU, causing a lot of extra cache traffic. > > > > OTOH, there's not much we can do about this short of going looking for > > better multi-CPU reference count implementations once we have time to > > worry about performance. > > Actually, you can just put it in non-cacheable memory, and the > penalty will only be paid by the CPU(s) doing the referencing. Yes. And you'll pay the penalty *all* the time. At least when the ping-pong is going on, there will be times when you'll hit the counter valid in your own cache. Marking it uncacheable (or even write-back cacheable) is worse. > Still, for a very large number of CPUs, this would work fine > for all but frequently contended objects. Er. We're talking about an object which is susceptible to being *very* frequently contended. > I think that it is making more and more sense to lock interrupts > to a single CPU. No, it's not. Stop this nonsense. It's not even practical on some of the platforms we're looking at. > What happens if you write to a page that's marked non-cachable > on the CPU on which you are running, but cacheable on another > CPU? Does it do the right thing, and update the cache on the > caching CPU? Er, what are you smoking Terry? You never 'update' the cache on another processor; the other processor snoops your cache/memory activity and invalidates its own cache based on your broadcasts. > If so, locking the interrupt processing for each > card to a particular CPU could be very worthwhile, since you > would never take the hit, unless you were doing something > extraordinary. With the way our I/O structure is currently laid out, this blows because you end up serialising everything. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 21:33:12 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 21:33:10 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mass.osd.bsdi.com (adsl-63-202-176-64.dsl.snfc21.pacbell.net [63.202.176.64]) by hub.freebsd.org (Postfix) with ESMTP id C437E37B400 for ; Thu, 7 Dec 2000 21:33:09 -0800 (PST) Received: from mass.osd.bsdi.com (localhost [127.0.0.1]) by mass.osd.bsdi.com (8.11.0/8.11.1) with ESMTP id eB85gVN00523 for ; Thu, 7 Dec 2000 21:42:31 -0800 (PST) (envelope-from msmith@mass.osd.bsdi.com) Message-Id: <200012080542.eB85gVN00523@mass.osd.bsdi.com> X-Mailer: exmh version 2.1.1 10/15/1999 To: smp@FreeBSD.ORG Subject: Re: Netgraph and SMP In-reply-to: Your message of "Thu, 07 Dec 2000 21:33:27 PST." <200012080533.eB85XRN00458@mass.osd.bsdi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 07 Dec 2000 21:42:31 -0800 From: Mike Smith Sender: msmith@mass.osd.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > What happens if you write to a page that's marked non-cachable > > on the CPU on which you are running, but cacheable on another > > CPU? Does it do the right thing, and update the cache on the > > caching CPU? > > Er, what are you smoking Terry? You never 'update' the cache on another > processor; the other processor snoops your cache/memory activity and > invalidates its own cache based on your broadcasts. I should have added "if you're lucky" to this. Some platforms don't even go that far. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Thu Dec 7 23:51:20 2000 From owner-freebsd-smp@FreeBSD.ORG Thu Dec 7 23:51:18 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from internal.mail.demon.net (internal.mail.demon.net [193.195.224.3]) by hub.freebsd.org (Postfix) with ESMTP id 06CC737B401 for ; Thu, 7 Dec 2000 23:51:17 -0800 (PST) Received: from gti.noc.demon.net (gti.noc.demon.net [195.11.55.101]) by internal.mail.demon.net with ESMTP id HAA15586; Fri, 8 Dec 2000 07:51:15 GMT Received: (from jguard@localhost) by gti.noc.demon.net (8.8.8/8.8.8) id HAA04677; Fri, 8 Dec 2000 07:51:11 GMT Date: Fri, 8 Dec 2000 07:51:11 +0000 From: James Guard To: Peter Wemm Cc: smp@FreeBSD.ORG Subject: Re: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fwd) Message-ID: <20001208075111.A4595@demon.net> References: <200012060656.eB66twJ29960@mobile.wemm.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200012060656.eB66twJ29960@mobile.wemm.org>; from peter@netplex.com.au on Tue, Dec 05, 2000 at 10:55:58PM -0800 Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Hi, On Tue, Dec 05, 2000 at 10:55:58PM -0800, Peter Wemm wrote: > FYI: I have seen a few folks with compaq DL360's post on the list.. I'd > appreciate a few more testers. This should backport to RELENG_4 trivially. > > http://people.freebsd.org/~peter/compaq.diff > :) I can also confirm that this allows us to boot a DL360 and see both CPU's. Everything seems to be working well. Thanks, James -- James Guard Senior Systems Administrator Interactive Services Demon Internet, Demon@THUS http://www.games.demon.net/ http://www.demon.net/ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 3:15:45 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 03:15:43 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mobile.wemm.org (adsl-64-163-195-99.dsl.snfc21.pacbell.net [64.163.195.99]) by hub.freebsd.org (Postfix) with ESMTP id 2D38F37B401; Fri, 8 Dec 2000 03:15:43 -0800 (PST) Received: from netplex.com.au (peter@localhost [127.0.0.1]) by mobile.wemm.org (8.11.1/8.11.1) with ESMTP id eB8BFb630929; Fri, 8 Dec 2000 03:15:38 -0800 (PST) (envelope-from peter@netplex.com.au) Message-Id: <200012081115.eB8BFb630929@mobile.wemm.org> X-Mailer: exmh version 2.2 06/23/2000 with nmh-1.0.4 To: John Baldwin Cc: The Hermit Hacker , smp@FreeBSD.ORG, IWAIZAKO Takahiro Subject: Re: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fw In-Reply-To: Date: Fri, 08 Dec 2000 03:15:37 -0800 From: Peter Wemm Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org John Baldwin wrote: > > On 07-Dec-00 The Hermit Hacker wrote: > > > > fixes me too on the IBM 7100 Quad: > > > > APIC_IO: Testing 8254 interrupt delivery > > APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpi n > > 2 > > APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 > > > > now, since both myself and IWAIZAKO are getting the exact same error, is > > it actually broken? :) this is Compaq vs IBM ... > > They are both broken. :) This definitely is a workaround for some broken > motherboards out there, not a case of us not handling a set of non-broken > motherboards. :) > > > Any chance of getting this patch into the 4.x tree? :) > > Peter? Done.. Cheers, -Peter -- Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au "All of this is for nothing if we don't go to the stars" - JMS/B5 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 5:11:54 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 05:11:52 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from thelab.hub.org (CDR22-166.accesscable.net [24.138.22.166]) by hub.freebsd.org (Postfix) with ESMTP id DBEB837B400; Fri, 8 Dec 2000 05:11:48 -0800 (PST) Received: from localhost (scrappy@localhost) by thelab.hub.org (8.11.1/8.11.1) with ESMTP id eB8DAwS38559; Fri, 8 Dec 2000 09:10:58 -0400 (AST) (envelope-from scrappy@hub.org) X-Authentication-Warning: thelab.hub.org: scrappy owned process doing -bs Date: Fri, 8 Dec 2000 09:10:58 -0400 (AST) From: The Hermit Hacker To: Peter Wemm Cc: John Baldwin , smp@FreeBSD.ORG, IWAIZAKO Takahiro Subject: Re: Compaq DL360; cvs commit: src/sys/i386/i386 mp_machdep.c (fw In-Reply-To: <200012081115.eB8BFb630929@mobile.wemm.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Great, thanks ... now if we could just convince "the big boys" that they are broken *grin* On Fri, 8 Dec 2000, Peter Wemm wrote: > John Baldwin wrote: > > > > On 07-Dec-00 The Hermit Hacker wrote: > > > > > > fixes me too on the IBM 7100 Quad: > > > > > > APIC_IO: Testing 8254 interrupt delivery > > > APIC_IO: Broken MP table detected: 8254 is not connected to IOAPIC #0 intpi > n > > > 2 > > > APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0 > > > > > > now, since both myself and IWAIZAKO are getting the exact same error, is > > > it actually broken? :) this is Compaq vs IBM ... > > > > They are both broken. :) This definitely is a workaround for some broken > > motherboards out there, not a case of us not handling a set of non-broken > > motherboards. :) > > > > > Any chance of getting this patch into the 4.x tree? :) > > > > Peter? > > Done.. > > Cheers, > -Peter > -- > Peter Wemm - peter@FreeBSD.org; peter@yahoo-inc.com; peter@netplex.com.au > "All of this is for nothing if we don't go to the stars" - JMS/B5 > > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 12:52:39 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 12:52:35 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp03.primenet.com (smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (Postfix) with ESMTP id 52DBD37B400; Fri, 8 Dec 2000 12:52:35 -0800 (PST) Received: (from daemon@localhost) by smtp03.primenet.com (8.9.3/8.9.3) id NAA07199; Fri, 8 Dec 2000 13:50:18 -0700 (MST) Received: from usr01.primenet.com(206.165.6.201) via SMTP by smtp03.primenet.com, id smtpdAAASXaW_n; Fri Dec 8 13:50:08 2000 Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id NAA22447; Fri, 8 Dec 2000 13:52:20 -0700 (MST) From: Terry Lambert Message-Id: <200012082052.NAA22447@usr01.primenet.com> Subject: Re: Netgraph and SMP To: msmith@FreeBSD.ORG (Mike Smith) Date: Fri, 8 Dec 2000 20:52:20 +0000 (GMT) Cc: tlambert@primenet.com (Terry Lambert), smp@FreeBSD.ORG In-Reply-To: <200012080533.eB85XRN00458@mass.osd.bsdi.com> from "Mike Smith" at Dec 07, 2000 09:33:27 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr01.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Actually, you can just put it in non-cacheable memory, and the > > penalty will only be paid by the CPU(s) doing the referencing. > > Yes. And you'll pay the penalty *all* the time. At least when the > ping-pong is going on, there will be times when you'll hit the counter > valid in your own cache. Marking it uncacheable (or even write-back > cacheable) is worse. The absolute worst thing you can do on a multiprocessor system is contend shared resources, either stalling another CPU or causing a cache invalidation. > > Still, for a very large number of CPUs, this would work fine > > for all but frequently contended objects. > > Er. We're talking about an object which is susceptible to being *very* > frequently contended. Right. Which is why you break the contention domain so that it is _not_ contending between CPUs. That way, only one CPU will pay the penalty. In the UP case, you can decide to not mark the page non-cacheable. > > I think that it is making more and more sense to lock interrupts > > to a single CPU. > > No, it's not. Stop this nonsense. It's not even practical on some of > the platforms we're looking at. NT does it on every platform on which it runs. It significantly beat both Linux and FreeBSD in the Ziff Davis benchmarks you and Jordan attended, using this configuration. For the platforms where it's not possible, I agree: you eat the synchronization overhead. BTW: aren't some of these platforms MEI, and not MESI? > > What happens if you write to a page that's marked non-cachable > > on the CPU on which you are running, but cacheable on another > > CPU? Does it do the right thing, and update the cache on the > > caching CPU? > > Er, what are you smoking Terry? You never 'update' the cache on another > processor; the other processor snoops your cache/memory activity and > invalidates its own cache based on your broadcasts. Let me explain the model: You mark the page cacheable on the processor that will contend the resource at interrupt time, and you make it uncacheable on other processors. The question is whether the write through is immediate or delayed (if delayed, the main memory value could be incorrect when examined by another CPU), and whether a write to main memory by a CPU without it cached will result in an proper invalidation in the CPU that has it cached (if so, then the approach will work). What this gives you is no inter-CPU contention, unless the main memory location is written by a processor that is not the interrupt processor for the lock for the driver being held. In that case, the only invalidation is against a single CPU, not multiple CPUs. This isn't necessarily a strategy limited to locked interrupts. By allocating lock regions in cache line lengths, you can then practically guarantee that, on a heavily loaded system, the invalidation triggered by a CPU will, at most, invalidate the cache line stored in _one_ other CPU (the last one prior to take to the interrupt, assuming that it is moving around). On a less heavily loaded system, the cache line for the lock region may be valid in multiple CPUs (not having been recycled), in which case you will take additional invalidation overhead. But doing so is less problematic, since you can afford the overhead when the system is less heavily loaded. This is really a "virtually non-cacheable" approach. If you can lock specific interrupts to a particular CPU, as NT locked one network card per CPU in the Ziff-Davis tests, then you achieve the same thing: the contended region is not ever referenced by the other CPUs, except under exceptional conditions, like driver unload, and it is effectively not cached on them as a result, even if the pages are marked cacheable. Does that make more sense? > > If so, locking the interrupt processing for each > > card to a particular CPU could be very worthwhile, since you > > would never take the hit, unless you were doing something > > extraordinary. > > With the way our I/O structure is currently laid out, this blows because > you end up serialising everything. Not everything; just interrupts for a single card: the locking would occur at an interrupt granularity; I'm not talking about ASMP here. You also only end up serializing them to a single CPU; as long as your load is reasonably distributed, the CPU won't be doing other work, at the time. Also the locking need not be literal: it could be nothing more than a "strong affinity", requiring extra effort to implement any migration. This is, I believe, how NT does it. PS: Considering all this, it makes sense to me to perhaps consider ensuring that a disk interrupt for a DMA completion be handled on the CPU responsible for the network card that will be sending the data that the completion signals availability on. NetWare does something similar, in that its threads are based on voluntary, not involuntary preemption. Admitted, this means true ASMP, but it's hard to argue with NetWare's file server performance... Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 12:56: 6 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 12:56:01 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (Postfix) with ESMTP id D0EB537B400; Fri, 8 Dec 2000 12:56:00 -0800 (PST) Received: (from daemon@localhost) by smtp02.primenet.com (8.9.3/8.9.3) id NAA10971; Fri, 8 Dec 2000 13:51:21 -0700 (MST) Received: from usr01.primenet.com(206.165.6.201) via SMTP by smtp02.primenet.com, id smtpdAAA3_aGsv; Fri Dec 8 13:51:10 2000 Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id NAA22567; Fri, 8 Dec 2000 13:55:45 -0700 (MST) From: Terry Lambert Message-Id: <200012082055.NAA22567@usr01.primenet.com> Subject: Re: Netgraph and SMP To: msmith@FreeBSD.ORG (Mike Smith) Date: Fri, 8 Dec 2000 20:55:44 +0000 (GMT) Cc: smp@FreeBSD.ORG In-Reply-To: <200012080542.eB85gVN00523@mass.osd.bsdi.com> from "Mike Smith" at Dec 07, 2000 09:42:31 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr01.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Er, what are you smoking Terry? You never 'update' the cache on another > > processor; the other processor snoops your cache/memory activity and > > invalidates its own cache based on your broadcasts. > > I should have added "if you're lucky" to this. Some platforms don't even > go that far. The BeBox is an example of a box without an L2 cachem and using the L2 cache control lines in order to implement MEI coherency for a maximum of two CPUs. You can actually get zero cycle bus arbitration to work with multiple MIPS processors (Phil Neiswanger initiated some work on this; I helped with the arbitration algorithm), also giving MEI coherency. I understand the limitations; please see my other posting. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 13:21:37 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 13:21:36 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 2C9CF37B400; Fri, 8 Dec 2000 13:21:35 -0800 (PST) Received: from berserker.bsdi.com (cp@localhost.bsdi.com [127.0.0.1]) by berserker.bsdi.com (8.11.1/8.9.3) with ESMTP id eB8LLSQ07456; Fri, 8 Dec 2000 14:21:28 -0700 (MST) (envelope-from cp@berserker.bsdi.com) Message-Id: <200012082121.eB8LLSQ07456@berserker.bsdi.com> To: Mike Smith Cc: smp@FreeBSD.ORG Subject: Re: Netgraph and SMP In-reply-to: Your message of "Thu, 07 Dec 2000 21:42:31 PST." <200012080542.eB85gVN00523@mass.osd.bsdi.com> From: Chuck Paterson Date: Fri, 08 Dec 2000 14:21:28 -0700 Sender: cp@berserker.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org For uses such as barriers for loading and unloading it is possible to have the counters and entry barriers all PCPU. You can then use more complex mechanisms to set the low level barrier and interrogate the counters. Terry ->>may<<- view this as another way of doing what he is suggesting. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 13:24:54 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 13:24:52 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from critter.freebsd.dk (flutter.freebsd.dk [212.242.40.147]) by hub.freebsd.org (Postfix) with ESMTP id A80E637B400; Fri, 8 Dec 2000 13:24:51 -0800 (PST) Received: from critter (localhost [127.0.0.1]) by critter.freebsd.dk (8.11.1/8.11.1) with ESMTP id eB8LOkL80665; Fri, 8 Dec 2000 22:24:46 +0100 (CET) (envelope-from phk@critter.freebsd.dk) To: Chuck Paterson Cc: Mike Smith , smp@FreeBSD.ORG Subject: Re: Netgraph and SMP In-Reply-To: Your message of "Fri, 08 Dec 2000 14:21:28 MST." <200012082121.eB8LLSQ07456@berserker.bsdi.com> Date: Fri, 08 Dec 2000 22:24:46 +0100 Message-ID: <80663.976310686@critter> From: Poul-Henning Kamp Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In message <200012082121.eB8LLSQ07456@berserker.bsdi.com>, Chuck Paterson write s: > >For uses such as barriers for loading and unloading it is >possible to have the counters and entry barriers all PCPU. You can then >use more complex mechanisms to set the low level barrier and interrogate >the counters. Terry ->>may<<- view this as another way of doing >what he is suggesting. The thing that has me worried here is that using locking (as opposed to atomic ops) in netgraph means that will expose netgraph paths to heavy-duty locking synchronism, since TCP, UDP, IP, Mbuf will also use a (separate) locking domain. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 13:32:11 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 13:32:08 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp05.primenet.com (smtp05.primenet.com [206.165.6.135]) by hub.freebsd.org (Postfix) with ESMTP id DFD0037B400; Fri, 8 Dec 2000 13:32:07 -0800 (PST) Received: (from daemon@localhost) by smtp05.primenet.com (8.9.3/8.9.3) id OAA15945; Fri, 8 Dec 2000 14:28:41 -0700 (MST) Received: from usr01.primenet.com(206.165.6.201) via SMTP by smtp05.primenet.com, id smtpdAAAlqaahF; Fri Dec 8 14:28:36 2000 Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id OAA23873; Fri, 8 Dec 2000 14:31:59 -0700 (MST) From: Terry Lambert Message-Id: <200012082131.OAA23873@usr01.primenet.com> Subject: Re: Netgraph and SMP To: cp@bsdi.com (Chuck Paterson) Date: Fri, 8 Dec 2000 21:31:59 +0000 (GMT) Cc: msmith@FreeBSD.ORG (Mike Smith), smp@FreeBSD.ORG In-Reply-To: <200012082121.eB8LLSQ07456@berserker.bsdi.com> from "Chuck Paterson" at Dec 08, 2000 02:21:28 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr01.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > For uses such as barriers for loading and unloading it is > possible to have the counters and entry barriers all PCPU. You can then > use more complex mechanisms to set the low level barrier and interrogate > the counters. Terry ->>may<<- view this as another way of doing > what he is suggesting. Yes. Specifically, it is not necessary to hold a mutex, only a spinlock, for the count adjust; you need only verify that the count is > 1, and the mutex is _not_ held. The negative logic saves you from the invalidation cycle on the shared resource (the mutex). The mutex need only be held when the driver is to be unloaded, or during loading, after wiring in, but prior to initialization. When doing this, you acquire the mutex before incrementing the hold count. For a non-zero hold count, you spin until it is zero, and deal with the unload. This means that the hold count and the mutex have to be associated with the slot for the driver, and not part of the loaded/unloaded data area, but it's well worth the effort. The other strategies having to do with caching are just gravy, in that they reduce the interprocessor contention considerably. You can go even further by locking interupts to cards, but that isn't necesary to get ~80% of the available win, anyway. This approach should be light enough to satisfy Julian, as well. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 13:51:57 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 13:51:54 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp10.phx.gblx.net (smtp10.phx.gblx.net [206.165.6.140]) by hub.freebsd.org (Postfix) with ESMTP id 8D70137B400; Fri, 8 Dec 2000 13:51:54 -0800 (PST) Received: (from daemon@localhost) by smtp10.phx.gblx.net (8.9.3/8.9.3) id OAA37580; Fri, 8 Dec 2000 14:51:52 -0700 Received: from usr01.primenet.com(206.165.6.201) via SMTP by smtp10.phx.gblx.net, id smtpdu0XuUa; Fri Dec 8 14:51:44 2000 Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id OAA24586; Fri, 8 Dec 2000 14:51:41 -0700 (MST) From: Terry Lambert Message-Id: <200012082151.OAA24586@usr01.primenet.com> Subject: Re: Netgraph and SMP To: phk@critter.freebsd.dk (Poul-Henning Kamp) Date: Fri, 8 Dec 2000 21:51:41 +0000 (GMT) Cc: cp@bsdi.com (Chuck Paterson), msmith@FreeBSD.ORG (Mike Smith), smp@FreeBSD.ORG In-Reply-To: <80663.976310686@critter> from "Poul-Henning Kamp" at Dec 08, 2000 10:24:46 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr01.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > >For uses such as barriers for loading and unloading it is > >possible to have the counters and entry barriers all PCPU. You can then > >use more complex mechanisms to set the low level barrier and interrogate > >the counters. Terry ->>may<<- view this as another way of doing > >what he is suggesting. > > The thing that has me worried here is that using locking (as opposed > to atomic ops) in netgraph means that will expose netgraph paths to > heavy-duty locking synchronism, since TCP, UDP, IP, Mbuf will also > use a (separate) locking domain. It ought to reference on entry to netgraph; this will let it avoid locks for everything but adjusting the reference count, and won't damage reentrancy. It would mean stalling all of netgraph when loading or unloading a module, or establishing a connection or disconnecting nodes, though, but this might be OK. Sort of a BGL with multiple users/single manipulator for Netgraph. I think it's overly complicated, but you could also support the idea of per CPU connectedness, which would build the graph out on each CPU, making the relationship between nodes a per CPU thing. This would mean that you would not need to contend to build a graph per CPU, but that you would have to duplicate the build on each CPU. This would reduce to dealing with the lock only on load and unload (again). Each graph would need to set pointers to a shared state data object, though, since they are effectively the same objects with different function pointer linkages. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 14: 7:33 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 14:07:30 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from fw.wintelcom.net (ns1.wintelcom.net [209.1.153.20]) by hub.freebsd.org (Postfix) with ESMTP id 4464137B400; Fri, 8 Dec 2000 14:07:30 -0800 (PST) Received: (from bright@localhost) by fw.wintelcom.net (8.10.0/8.10.0) id eB8M4sm27379; Fri, 8 Dec 2000 14:04:54 -0800 (PST) Date: Fri, 8 Dec 2000 14:04:54 -0800 From: Alfred Perlstein To: Terry Lambert Cc: Poul-Henning Kamp , Chuck Paterson , Mike Smith , smp@FreeBSD.ORG Subject: Re: Netgraph and SMP Message-ID: <20001208140454.S16205@fw.wintelcom.net> References: <80663.976310686@critter> <200012082151.OAA24586@usr01.primenet.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200012082151.OAA24586@usr01.primenet.com>; from tlambert@primenet.com on Fri, Dec 08, 2000 at 09:51:41PM +0000 Sender: bright@fw.wintelcom.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org * Terry Lambert [001208 13:52] wrote: > > >For uses such as barriers for loading and unloading it is > > >possible to have the counters and entry barriers all PCPU. You can then > > >use more complex mechanisms to set the low level barrier and interrogate > > >the counters. Terry ->>may<<- view this as another way of doing > > >what he is suggesting. > > > > The thing that has me worried here is that using locking (as opposed > > to atomic ops) in netgraph means that will expose netgraph paths to > > heavy-duty locking synchronism, since TCP, UDP, IP, Mbuf will also > > use a (separate) locking domain. > > It ought to reference on entry to netgraph; this will let it > avoid locks for everything but adjusting the reference count, > and won't damage reentrancy. > > It would mean stalling all of netgraph when loading or unloading > a module, or establishing a connection or disconnecting nodes, > though, but this might be OK. > > Sort of a BGL with multiple users/single manipulator for Netgraph. > > > I think it's overly complicated, but you could also support the > idea of per CPU connectedness, which would build the graph out > on each CPU, making the relationship between nodes a per CPU > thing. This would mean that you would not need to contend to > build a graph per CPU, but that you would have to duplicate the > build on each CPU. This would reduce to dealing with the lock > only on load and unload (again). Each graph would need to set > pointers to a shared state data object, though, since they are > effectively the same objects with different function pointer > linkages. *hack* *hack* *hack* *hack* *hack* *hack* *hack* *hack* (me not you. :)) netgraph_locks[NCPU]; netgraph_refs[NCPU]; unload() { for (i = 0; i< NCPU; i++) if (!mutex_try(&netgraph_locks[i]) { i--; goto error; } else if (netgraph_refs[i] != 0) { goto error; } do_unload(); return (0); error: while (i--) mutex_unlock(&netgraph_locks[i]) return (EBUSY); } *hack* *hack* *hack* *hack* *hack* *hack* *hack* *hack* ? -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 14:41: 3 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 14:41:02 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id 07D0D37B400; Fri, 8 Dec 2000 14:41:01 -0800 (PST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id EA73C6AB6B; Sat, 9 Dec 2000 09:10:57 +1030 (CST) Date: Sat, 9 Dec 2000 09:10:57 +1030 From: Greg Lehey To: Jake Burkholder Cc: developers@lemis.com, FreeBSD SMP list Subject: Re: cvs commit: src/sys/sys ipl.h Message-ID: <20001209091057.H37171@wantadilla.lemis.com> References: <200012081039.eB8Ad2214309@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200012081039.eB8Ad2214309@freefall.freebsd.org>; from jake@FreeBSD.org on Fri, Dec 08, 2000 at 02:39:02AM -0800 Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Friday, 8 December 2000 at 2:39:02 -0800, Jake Burkholder wrote: > jake 2000/12/08 02:39:02 PST > > Modified files: > sys/sys ipl.h > Log: > Remove unused declarations for spending and sdelayed, and remove unused > #defines for bits corresponding to pending interrupts. Yay threads! I'm a bit concerned about the pace of change in this code. We have significant problems in the current kernel (for example, I haven't been able to complete a make world for a week now, because the system goes catatonic). We should be looking for these problems, difficult as it may be, not moving on. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 15: 8: 7 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 15:08:04 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from magnesium.net (toxic.magnesium.net [207.154.84.15]) by hub.freebsd.org (Postfix) with SMTP id 1825C37B6C5 for ; Fri, 8 Dec 2000 15:02:48 -0800 (PST) Received: (qmail 68665 invoked by uid 1142); 8 Dec 2000 23:02:47 -0000 Date: 8 Dec 2000 15:02:47 -0800 Date: Fri, 8 Dec 2000 15:02:40 -0800 From: Jason Evans To: Greg Lehey Cc: FreeBSD SMP list Subject: Re: cvs commit: src/sys/sys ipl.h Message-ID: <20001208150240.S2312@canonware.com> References: <200012081039.eB8Ad2214309@freefall.freebsd.org> <20001209091057.H37171@wantadilla.lemis.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20001209091057.H37171@wantadilla.lemis.com>; from grog@lemis.com on Sat, Dec 09, 2000 at 09:10:57AM +1030 Sender: jasone@magnesium.net Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Sat, Dec 09, 2000 at 09:10:57AM +1030, Greg Lehey wrote: > On Friday, 8 December 2000 at 2:39:02 -0800, Jake Burkholder wrote: > > jake 2000/12/08 02:39:02 PST > > > > Modified files: > > sys/sys ipl.h > > Log: > > Remove unused declarations for spending and sdelayed, and remove unused > > #defines for bits corresponding to pending interrupts. Yay threads! > > I'm a bit concerned about the pace of change in this code. We have > significant problems in the current kernel (for example, I haven't > been able to complete a make world for a week now, because the system > goes catatonic). We should be looking for these problems, difficult > as it may be, not moving on. Greg, "this code" is a bunch of unused cpp macros. How are you concerned that this might further increase instability? Just because there are existing known bugs doesn't mean that commits should unilaterally stop. Jason To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 15:15:30 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 15:15:28 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id C95CD37B400 for ; Fri, 8 Dec 2000 15:15:25 -0800 (PST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id 79EC26AB6B; Sat, 9 Dec 2000 09:45:21 +1030 (CST) Date: Sat, 9 Dec 2000 09:45:21 +1030 From: Greg Lehey To: Jason Evans Cc: FreeBSD SMP list Subject: Re: cvs commit: src/sys/sys ipl.h Message-ID: <20001209094521.D53111@wantadilla.lemis.com> References: <200012081039.eB8Ad2214309@freefall.freebsd.org> <20001209091057.H37171@wantadilla.lemis.com> <20001208150240.S2312@canonware.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20001208150240.S2312@canonware.com>; from jasone@canonware.com on Fri, Dec 08, 2000 at 03:02:40PM -0800 Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Friday, 8 December 2000 at 15:02:40 -0800, Jason Evans wrote: > On Sat, Dec 09, 2000 at 09:10:57AM +1030, Greg Lehey wrote: >> On Friday, 8 December 2000 at 2:39:02 -0800, Jake Burkholder wrote: >>> jake 2000/12/08 02:39:02 PST >>> >>> Modified files: >>> sys/sys ipl.h >>> Log: >>> Remove unused declarations for spending and sdelayed, and remove unused >>> #defines for bits corresponding to pending interrupts. Yay threads! >> >> I'm a bit concerned about the pace of change in this code. We have >> significant problems in the current kernel (for example, I haven't >> been able to complete a make world for a week now, because the system >> goes catatonic). We should be looking for these problems, difficult >> as it may be, not moving on. > > Greg, "this code" is a bunch of unused cpp macros. How are you > concerned that this might further increase instability? It's certainly not increasing the stability. > Just because there are existing known bugs doesn't mean that commits > should unilaterally stop. No, but as far as I can tell nobody's looking for the ones we have. Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 16: 4: 5 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 16:04:02 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mail.interware.hu (mail.interware.hu [195.70.32.130]) by hub.freebsd.org (Postfix) with ESMTP id 5817637B6A6; Fri, 8 Dec 2000 16:04:00 -0800 (PST) Received: from monrovia-14.budapest.interware.hu ([195.70.53.206] helo=elischer.org) by mail.interware.hu with esmtp (Exim 3.16 #1 (Debian)) id 144XUb-0003WN-00; Sat, 09 Dec 2000 01:03:57 +0100 Sender: julian@FreeBSD.ORG Message-ID: <3A3176D2.C3587387@elischer.org> Date: Fri, 08 Dec 2000 16:03:30 -0800 From: Julian Elischer X-Mailer: Mozilla 4.7 [en] (X11; U; FreeBSD 5.0-CURRENT i386) X-Accept-Language: en, hu MIME-Version: 1.0 To: Poul-Henning Kamp Cc: Chuck Paterson , Mike Smith , smp@FreeBSD.ORG Subject: Re: Netgraph and SMP References: <80663.976310686@critter> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Poul-Henning Kamp wrote: > > In message <200012082121.eB8LLSQ07456@berserker.bsdi.com>, Chuck Paterson write > s: > > > >For uses such as barriers for loading and unloading it is > >possible to have the counters and entry barriers all PCPU. You can then > >use more complex mechanisms to set the low level barrier and interrogate > >the counters. Terry ->>may<<- view this as another way of doing > >what he is suggesting. > > The thing that has me worried here is that using locking (as opposed > to atomic ops) in netgraph means that will expose netgraph paths to > heavy-duty locking synchronism, since TCP, UDP, IP, Mbuf will also > use a (separate) locking domain. My aim is to have as little locking as possible. (You should know since I've been CCing you for some of the discussions archie and I were having) The difficulty comes with trying to not fall off the edge of the world when a node or link is reconfigured while data is transiting through the system. At the moment I'm using reference counts and VALID/INVALID flags to stop things from disapearing out from under running code, but the boundary conditions are really difficult. It's easy to stop a node from going away while the packeet is fairly and squarly in teh node's control, but the problem occurs when a packet is transitionning from one node to another. There are race conditions all over the place. I think that some simple spinlocks can remove those race conditions but I'd like to use locks that allow more than one packet at a time to transition through a node or graph. This implies a reader/writer lock, as Mike Smith pointed out. Anyhow it's all theoretical at the moment as I'm still only drawing on paper napkins at this stage. (BTW any productive suggestions would be welcome) > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk@FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-smp" in the body of the message -- __--_|\ Julian Elischer / \ julian@elischer.org ( OZ ) World tour 2000 ---> X_.---._/ presently in: Budapest v To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 21:30:20 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 21:30:18 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (Postfix) with ESMTP id 5208237B400 for ; Fri, 8 Dec 2000 21:30:18 -0800 (PST) Received: (from daemon@localhost) by smtp02.primenet.com (8.9.3/8.9.3) id WAA15976; Fri, 8 Dec 2000 22:25:38 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp02.primenet.com, id smtpdAAARYaOgF; Fri Dec 8 22:25:28 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id WAA06798; Fri, 8 Dec 2000 22:30:02 -0700 (MST) From: Terry Lambert Message-Id: <200012090530.WAA06798@usr08.primenet.com> Subject: Re: cvs commit: src/sys/sys ipl.h To: grog@lemis.com (Greg Lehey) Date: Sat, 9 Dec 2000 05:30:01 +0000 (GMT) Cc: jasone@canonware.com (Jason Evans), FreeBSD-smp@FreeBSD.ORG (FreeBSD SMP list) In-Reply-To: <20001209094521.D53111@wantadilla.lemis.com> from "Greg Lehey" at Dec 09, 2000 09:45:21 AM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr08.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > Just because there are existing known bugs doesn't mean that commits > > should unilaterally stop. > > No, but as far as I can tell nobody's looking for the ones we have. That's easy: if there is a version without the problem, and a later one with the problem, if the person who did the commit of the later version doesn't fix it, back the change out. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 22: 8: 6 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 22:08:04 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from zibbi.icomtek.csir.co.za (zibbi.icomtek.csir.co.za [146.64.24.58]) by hub.freebsd.org (Postfix) with ESMTP id 996AF37B400; Fri, 8 Dec 2000 22:08:00 -0800 (PST) Received: (from jhay@localhost) by zibbi.icomtek.csir.co.za (8.11.0/8.11.0) id eB967Yr06047; Sat, 9 Dec 2000 08:07:34 +0200 (SAT) (envelope-from jhay) From: John Hay Message-Id: <200012090607.eB967Yr06047@zibbi.icomtek.csir.co.za> Subject: Re: Hangs during 'make world -j4' on recent current In-Reply-To: <20001208124330.Z27667@wantadilla.lemis.com> from Greg Lehey at "Dec 8, 2000 12:43:30 pm" To: grog@lemis.com (Greg Lehey) Date: Sat, 9 Dec 2000 08:07:34 +0200 (SAT) Cc: FreeBSD-current@FreeBSD.ORG (FreeBSD current users), FreeBSD-smp@FreeBSD.ORG (FreeBSD SMP list) X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: jhay@zibbi.icomtek.csir.co.za Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > For over a week now, I have been unable to complete a 'make world' on > my -CURRENT box if I specify -j4. The system hangs and is completely > unresponsive. This is a dual Celeron and an Abit BP6 motherboard. As > far as I can tell, nobody knows what's causing this, nor even how to > attack the problem. I'd like to solicit feedback about the extent of > the problem, the possible causes, and how to debug it. I have been building releases with WORLD_FLAGS=-j4 successfully on my SMP box with a Dec 1 kernel for the past week. Yesterday I upgraded the kernel and with the new kernel did a make world -j4 which completed with no problems. And afterwards a make release with WORLD_FLAGS=-j4 also finished with no problems. So -current isn't totally broken. It might be timing related. My machine is an old dual 266MHz PII. John -- John Hay -- John.Hay@icomtek.csir.co.za To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 22:21:54 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 22:21:52 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from sharmas.dhs.org (c62443-a.frmt1.sfba.home.com [24.0.69.165]) by hub.freebsd.org (Postfix) with ESMTP id 5754237B400; Fri, 8 Dec 2000 22:21:52 -0800 (PST) Received: (from adsharma@localhost) by sharmas.dhs.org (8.9.3/8.9.3) id WAA28099; Fri, 8 Dec 2000 22:20:15 -0800 Date: Fri, 8 Dec 2000 22:20:15 -0800 From: Arun Sharma Message-Id: <200012090620.WAA28099@sharmas.dhs.org> To: jhb@FreeBSD.ORG Cc: smp@FreeBSD.ORG Subject: Re: Userland atomic assignments In-Reply-To: References: Reply-To: arun@sharmas.dhs.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 7 Dec 2000 23:26:10 +0100, John Baldwin wrote: > a = b; should be atomic I think. If its not, then large portions of the kernel > break, IIRC. a = b, may be atomic depending on the types of a and b. On 32 bit machines, everything less than 32 bits should be atomic. However, the complexity comes from memory ordering issues. All of x86 boxes ensure sequential consistency. Some subset of Alpha boxes [1] provide a weak ordering model as a way to improve performance. On such boxes, after executing: a = b c = d (c == d) does not imply (a == b). You need to have an explicit memory barrier in between. -Arun [1] Can any Alpha gurus comment on this ? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 22:31: 0 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 22:30:58 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from sharmas.dhs.org (c62443-a.frmt1.sfba.home.com [24.0.69.165]) by hub.freebsd.org (Postfix) with ESMTP id C8D1B37B400 for ; Fri, 8 Dec 2000 22:30:57 -0800 (PST) Received: (from adsharma@localhost) by sharmas.dhs.org (8.9.3/8.9.3) id WAA28103; Fri, 8 Dec 2000 22:29:19 -0800 Date: Fri, 8 Dec 2000 22:29:19 -0800 From: Arun Sharma Message-Id: <200012090629.WAA28103@sharmas.dhs.org> To: eischen@vigrid.com Cc: smp@freebsd.org Subject: Re: Userland atomic assignments In-Reply-To: References:

Reply-To: adsharma@sharmas.dhs.org Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On 7 Dec 2000 23:23:18 +0100, Daniel Eischen wrote: > > I don't need cmpset operations, just atomic int32 and ptr assignments. > The latter would be good enough to let me walk a simple list without > having to take a lock. > I thought the following URL might be useful to you: http://www.cs.rochester.edu/u/www/u/scott/synchronization/home.html Especially, the concurrent queue algorithms. -Arun To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 22:56: 2 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 22:56:00 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from smtp04.primenet.com (smtp04.primenet.com [206.165.6.134]) by hub.freebsd.org (Postfix) with ESMTP id 41D0837B400 for ; Fri, 8 Dec 2000 22:56:00 -0800 (PST) Received: (from daemon@localhost) by smtp04.primenet.com (8.9.3/8.9.3) id XAA20291; Fri, 8 Dec 2000 23:51:53 -0700 (MST) Received: from usr08.primenet.com(206.165.6.208) via SMTP by smtp04.primenet.com, id smtpdAAA1wa4IN; Fri Dec 8 23:51:48 2000 Received: (from tlambert@localhost) by usr08.primenet.com (8.8.5/8.8.5) id XAA08475; Fri, 8 Dec 2000 23:55:28 -0700 (MST) From: Terry Lambert Message-Id: <200012090655.XAA08475@usr08.primenet.com> Subject: Re: Userland atomic assignments To: adsharma@sharmas.dhs.org Date: Sat, 9 Dec 2000 06:55:27 +0000 (GMT) Cc: eischen@vigrid.com, smp@FreeBSD.ORG In-Reply-To: <200012090629.WAA28103@sharmas.dhs.org> from "Arun Sharma" at Dec 08, 2000 10:29:19 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: tlambert@usr08.primenet.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > > I don't need cmpset operations, just atomic int32 and ptr assignments. > > The latter would be good enough to let me walk a simple list without > > having to take a lock. > > I thought the following URL might be useful to you: > > http://www.cs.rochester.edu/u/www/u/scott/synchronization/home.html > > Especially, the concurrent queue algorithms. The preemption-safe reader-writer locks seem to be excellent, and just what the doctor ordered for Julian's work on netgraph... Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 23:12: 5 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 23:12:02 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from wantadilla.lemis.com (wantadilla.lemis.com [192.109.197.80]) by hub.freebsd.org (Postfix) with ESMTP id C1AF637B400; Fri, 8 Dec 2000 23:12:00 -0800 (PST) Received: by wantadilla.lemis.com (Postfix, from userid 1004) id 043EC6AB68; Sat, 9 Dec 2000 17:41:54 +1030 (CST) Date: Sat, 9 Dec 2000 17:41:53 +1030 From: Greg Lehey To: John Hay Cc: FreeBSD current users , FreeBSD SMP list Subject: Re: Hangs during 'make world -j4' on recent current Message-ID: <20001209174153.V53111@wantadilla.lemis.com> References: <20001208124330.Z27667@wantadilla.lemis.com> <200012090607.eB967Yr06047@zibbi.icomtek.csir.co.za> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <200012090607.eB967Yr06047@zibbi.icomtek.csir.co.za>; from jhay@icomtek.csir.co.za on Sat, Dec 09, 2000 at 08:07:34AM +0200 Organization: LEMIS, PO Box 460, Echunga SA 5153, Australia Phone: +61-8-8388-8286 Fax: +61-8-8388-8725 Mobile: +61-418-838-708 WWW-Home-Page: http://www.lemis.com/~grog X-PGP-Fingerprint: 6B 7B C3 8C 61 CD 54 AF 13 24 52 F8 6D A4 95 EF Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Saturday, 9 December 2000 at 8:07:34 +0200, John Hay wrote: >> For over a week now, I have been unable to complete a 'make world' on >> my -CURRENT box if I specify -j4. The system hangs and is completely >> unresponsive. This is a dual Celeron and an Abit BP6 motherboard. As >> far as I can tell, nobody knows what's causing this, nor even how to >> attack the problem. I'd like to solicit feedback about the extent of >> the problem, the possible causes, and how to debug it. > > I have been building releases with WORLD_FLAGS=-j4 successfully on my > SMP box with a Dec 1 kernel for the past week. Yesterday I upgraded the > kernel and with the new kernel did a make world -j4 which completed with > no problems. And afterwards a make release with WORLD_FLAGS=-j4 also > finished with no problems. So -current isn't totally broken. It might > be timing related. My machine is an old dual 266MHz PII. How many processors does your machine have? Greg -- Finger grog@lemis.com for PGP public key See complete headers for address and phone numbers To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Fri Dec 8 23:44:21 2000 From owner-freebsd-smp@FreeBSD.ORG Fri Dec 8 23:44:18 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from zibbi.icomtek.csir.co.za (zibbi.icomtek.csir.co.za [146.64.24.58]) by hub.freebsd.org (Postfix) with ESMTP id D640237B400; Fri, 8 Dec 2000 23:44:14 -0800 (PST) Received: (from jhay@localhost) by zibbi.icomtek.csir.co.za (8.11.0/8.11.0) id eB97hwW08236; Sat, 9 Dec 2000 09:43:58 +0200 (SAT) (envelope-from jhay) From: John Hay Message-Id: <200012090743.eB97hwW08236@zibbi.icomtek.csir.co.za> Subject: Re: Hangs during 'make world -j4' on recent current In-Reply-To: <20001209174153.V53111@wantadilla.lemis.com> from Greg Lehey at "Dec 9, 2000 05:41:53 pm" To: grog@lemis.com (Greg Lehey) Date: Sat, 9 Dec 2000 09:43:58 +0200 (SAT) Cc: jhay@icomtek.csir.co.za (John Hay), FreeBSD-current@FreeBSD.ORG (FreeBSD current users), FreeBSD-smp@FreeBSD.ORG (FreeBSD SMP list) X-Mailer: ELM [version 2.4ME+ PL54 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: jhay@zibbi.icomtek.csir.co.za Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > >> For over a week now, I have been unable to complete a 'make world' on > >> my -CURRENT box if I specify -j4. The system hangs and is completely > >> unresponsive. This is a dual Celeron and an Abit BP6 motherboard. As > >> far as I can tell, nobody knows what's causing this, nor even how to > >> attack the problem. I'd like to solicit feedback about the extent of > >> the problem, the possible causes, and how to debug it. > > > > I have been building releases with WORLD_FLAGS=-j4 successfully on my > > SMP box with a Dec 1 kernel for the past week. Yesterday I upgraded the > > kernel and with the new kernel did a make world -j4 which completed with > > no problems. And afterwards a make release with WORLD_FLAGS=-j4 also > > finished with no problems. So -current isn't totally broken. It might > > be timing related. My machine is an old dual 266MHz PII. ^^^^ > > How many processors does your machine have? 2 John -- John Hay -- John.Hay@icomtek.csir.co.za To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Dec 9 8: 2:44 2000 From owner-freebsd-smp@FreeBSD.ORG Sat Dec 9 08:02:41 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id F19D737B6A9; Sat, 9 Dec 2000 08:02:37 -0800 (PST) Received: from berserker.bsdi.com (cp@localhost.bsdi.com [127.0.0.1]) by berserker.bsdi.com (8.11.1/8.9.3) with ESMTP id eB9G2UH06249; Sat, 9 Dec 2000 09:02:30 -0700 (MST) (envelope-from cp@berserker.bsdi.com) Message-Id: <200012091602.eB9G2UH06249@berserker.bsdi.com> To: arun@sharmas.dhs.org Cc: jhb@FreeBSD.ORG, smp@FreeBSD.ORG Subject: Re: Userland atomic assignments In-reply-to: Your message of "Fri, 08 Dec 2000 22:20:15 PST." <200012090620.WAA28099@sharmas.dhs.org> From: Chuck Paterson Date: Sat, 09 Dec 2000 09:02:30 -0700 Sender: cp@berserker.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org } }However, the complexity comes from memory ordering issues. All of x86 boxes }ensure sequential consistency. } Sequenctial consistency from a single processor, which may well be what you meant. Writes from different processors to different cache lines are not ordered with respect to one another. Also writes from one processor are not at all ordered with respect to reads from another processor. What is guaranteed is that writes from a single processor will be ordered with respect to one another as viewed from any processor. Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Dec 9 8: 4:41 2000 From owner-freebsd-smp@FreeBSD.ORG Sat Dec 9 08:04:40 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 9A99F37B400 for ; Sat, 9 Dec 2000 08:04:39 -0800 (PST) Received: from berserker.bsdi.com (cp@localhost.bsdi.com [127.0.0.1]) by berserker.bsdi.com (8.11.1/8.9.3) with ESMTP id eB9G4UH06285; Sat, 9 Dec 2000 09:04:30 -0700 (MST) (envelope-from cp@berserker.bsdi.com) Message-Id: <200012091604.eB9G4UH06285@berserker.bsdi.com> To: Terry Lambert Cc: adsharma@sharmas.dhs.org, eischen@vigrid.com, smp@FreeBSD.ORG Subject: Re: Userland atomic assignments In-reply-to: Your message of "Sat, 09 Dec 2000 06:55:27 GMT." <200012090655.XAA08475@usr08.primenet.com> From: Chuck Paterson Date: Sat, 09 Dec 2000 09:04:30 -0700 Sender: cp@berserker.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org }The preemption-safe reader-writer locks seem to be excellent, }and just what the doctor ordered for Julian's work on netgraph... } } } Terry Lambert } terry@lambert.org Terry, I'm having trouble finding the example which doesn't use lock operations which is what we were trying to avoid. Could you put a more exact reference. Thanks Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Dec 9 8:21:36 2000 From owner-freebsd-smp@FreeBSD.ORG Sat Dec 9 08:21:34 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from berserker.bsdi.com (berserker.twistedbit.com [199.79.183.1]) by hub.freebsd.org (Postfix) with ESMTP id 1DE2A37B400; Sat, 9 Dec 2000 08:21:33 -0800 (PST) Received: from berserker.bsdi.com (cp@localhost.bsdi.com [127.0.0.1]) by berserker.bsdi.com (8.11.1/8.9.3) with ESMTP id eB9GGtH06408; Sat, 9 Dec 2000 09:16:55 -0700 (MST) (envelope-from cp@berserker.bsdi.com) Message-Id: <200012091616.eB9GGtH06408@berserker.bsdi.com> To: Julian Elischer Cc: Poul-Henning Kamp , Mike Smith , smp@FreeBSD.ORG Subject: Re: Netgraph and SMP In-reply-to: Your message of "Fri, 08 Dec 2000 16:03:30 PST." <3A3176D2.C3587387@elischer.org> From: Chuck Paterson Date: Sat, 09 Dec 2000 09:16:55 -0700 Sender: cp@berserker.bsdi.com Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Once again I would urge people to just make it work now and worry about making it fast later. In this case for now it really seems like we should just wrap it in a macro and use the lock manager, which is know to work. We can go back and fix performance when we have a stable platform and know we are just fixing the bugs we introduce with the speed optimizations. I would even more strongly urge people to not make architectural changes now. There are definately less expensive locks that can be fashioned for specific use profiles, such as almost always reader use. In my opinion we should 1) Get the system converted and stable 2) Make higher performance lock primitives for specific use profiles. 3) Look at making architectural changes to the pre-existing system. Chuck Chuck To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-smp" in the body of the message From owner-freebsd-smp Sat Dec 9 16:31:21 2000 From owner-freebsd-smp@FreeBSD.ORG Sat Dec 9 16:31:12 2000 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from libero.sunshine.ale (ppp-152-114.33-151.iol.it [151.33.114.152]) by hub.freebsd.org (Postfix) with ESMTP id C9D8337B400; Sat, 9 Dec 2000 16:31:10 -0800 (PST) Received: by libero.sunshine.ale (Postfix, from userid 1001) id 751675E9F; Sun, 10 Dec 2000 01:29:52 +0100 (CET) Date: Sun, 10 Dec 2000 01:29:52 +0100 From: Alessandro de Manzano To: freebsd-questions@freebsd.org Cc: freebsd-smp@freebsd.org Subject: USB and SMP Message-ID: <20001210012952.A343@libero.sunshine.ale> Reply-To: Alessandro de Manzano Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="YiEDa0DAkWCtVeE4" Content-Disposition: inline User-Agent: Mutt/1.2.5i X-Operating-System: FreeBSD 4.2-STABLE Sender: owner-freebsd-smp@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org --YiEDa0DAkWCtVeE4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello! I've a little technical problem and I would ask you for help :-) Situation: FreeBSD 4.2-STABLE (cvsupped few days ago) on a Microstar MS-6321 dual socket 370 motherboard. The CPUs are two Celeron 433 Mhz and the motherboard chipset is VIA VT82C694X (Apollo Pro 133A) with also a Promise ATA 100 dual channel controller (not used). I attached to this email my "dmesg" output. I noted that the USB subsystem get really crazy when running a SMP kernel. I've a Logitech Pilot Wheel Mouse USB that I used really fine on another 4.2-stable box, but on this one it is not usable. As you'll see from dmesg output, the USB controllers are correctly found, but seems not to work. But if I boot using a single CPU kernel then USB works fine. What could be ? Is this a known problem ? Here I include also the "usbdevs" output captured under SMP kernel: libero:(root)/root# usbdevs -v Controller /dev/usb0: addr 1: self powered, config 1, UHCI root hub(0x0000), VIA(0x0000), rev 0x0100 port 1 powered port 2 powered Controller /dev/usb1: addr 1: self powered, config 1, UHCI root hub(0x0000), VIA(0x0000), rev 0x0100 port 1 powered port 2 powered I guess that 0x0000 are not correct values for that parameters.. ;-) Please, could someone help me ? What I should do, besides disabling SMP ? ;-) Thanks a lot anyway! NOTE: please CC: the answer to me directly too, since I'm not yet subscribed to these lists! Thanks! -- bye! Alessandro de Manzano Milano, Italy aledema@iol.it --YiEDa0DAkWCtVeE4 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=dmesg Copyright (c) 1992-2000 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.2-STABLE #0: Sat Dec 9 19:15:36 CET 2000 root@libero.sunshine.ale:/usr/obj/usr/src/sys/LIBERO2 Timecounter "i8254" frequency 1193182 Hz CPU: Pentium II/Pentium II Xeon/Celeron (434.32-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x665 Stepping = 5 Features=0x183fbff real memory = 100663296 (98304K bytes) avail memory = 93990912 (91788K bytes) Programming 24 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 FreeBSD/SMP: Multiprocessor motherboard cpu0 (BSP): apic id: 0, version: 0x00040011, at 0xfee00000 cpu1 (AP): apic id: 1, version: 0x00040011, at 0xfee00000 io0 (APIC): apic id: 2, version: 0x00170011, at 0xfec00000 Preloaded elf kernel "kernel" at 0xc03b3000. Pentium Pro MTRR support enabled md0: Malloc disk npx0:

on motherboard
npx0: INT 16 interface
pcib0: on motherboard
pci0: on pcib0
pcib2: at device 1.0 on pci0
pci1: on pcib2
pci1: at 0.0 irq 16
isab0: at device 7.0 on pci0
isa0: on isab0
atapci0: port 0xc000-0xc00f at device 7.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
uhci0: port 0xc400-0xc41f irq 19 at device 7.2 on pci0
usb0: on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: port 0xc800-0xc81f irq 19 at device 7.3 on pci0
usb1: on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
pcm0: port 0xd400-0xd403,0xd000-0xd003,0xcc00-0xccff irq 18 at device 7.5 on pci0
atapci1: port 0xe800-0xe83f,0xe400-0xe403,0xe000-0xe007,0xdc00-0xdc03,0xd800-0xd807 mem 0xdc000000-0xdc01ffff irq 18 at device 12.0 on pci0
ata2: at 0xd800 on atapci1
ata3: at 0xe000 on atapci1
pci0: (vendor=0x104c, dev=0x8020) at 13.0 irq 19
xl0: <3Com 3c900-COMBO Etherlink XL> port 0xec00-0xec3f irq 17 at device 15.0 on pci0
xl0: Ethernet address: 00:10:4b:b0:38:07
xl0: selecting 10baseT transceiver, half duplex
pcib1: on motherboard
pci2: on pcib1
fdc0: at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: at port 0x60,0x64 on isa0
atkbd0: flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: irq 12 on atkbdc0
psm0: model IntelliMouse, device ID 3
vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ppc0: at port 0x378-0x37f irq 7 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppi0: on ppbus0
lpt0: on ppbus0
lpt0: Interrupt-driven port
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
IP packet filtering initialized, divert disabled, rule-based forwarding enabled, default to accept, logging limited to 100 packets/entry by default
DUMMYNET initialized (000608)
BRIDGE 990810, have 8 interfaces
-- index 1  type 6 phy 0 addrl 6 addr 00.10.4b.b0.38.07
SMP: AP CPU #1 Launched!
ad0: 9787MB [19885/16/63] at ata0-master UDMA66
acd0: CDROM at ata1-master using WDMA2
Mounting root from ufs:/dev/ad0s1a

--YiEDa0DAkWCtVeE4--


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message


From owner-freebsd-smp  Sat Dec  9 16:34:55 2000
From owner-freebsd-smp@FreeBSD.ORG  Sat Dec  9 16:34:53 2000
Return-Path: Delivered-To: freebsd-smp@freebsd.org
Received: from Athena.webmagix.net (athena.webmagix.net [212.189.235.2])
	by hub.freebsd.org (Postfix) with ESMTP id B392237B400
	for; Sat,  9 Dec 2000 16:34:51 -0800 (PST)
Received: from ns (c106231.upc-c.chello.nl [212.187.106.231])
	by Athena.webmagix.net (8.10.1/8.10.1) with SMTP id eBA0Yoo32574
	for; Sun, 10 Dec 2000 01:34:50 +0100
From: "Rick Jansen" To: Subject: MySQLd using just one CPU
Date: Sun, 10 Dec 2000 01:37:12 +0100
Message-ID: MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0)
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200
Importance: Normal
Sender: owner-freebsd-smp@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.org

Hi,

I have a problem with my dedicated MySQL server running on 4.2-STABLE. When
i stress the daemon to the max, and i look at top, the WCPU and CPU fields
are both around 99%. Which is what i want. BUT... Looking at the CPU States,
it says the machine is around 49% idle! This leads me to conclude that MySQL
is using just one processor. In Linux, there are a lot of mysqld processes
because of the different threadpackage (LinuxThreads instead of MIT). Is
there a threadpackage of this kind for FreeBSD aswell? I guess that would
use both cpu's..

What can i do about this? It's an awful waste of such a nice P3 733..

TIA

Rick Jansen


******************************************
  Server Administrator [Linux - FreeBSD]
 Websites: www.shellz.nl www.tweakers.net
              ICQ#: 3716519
E-mail: Rick@Tweakers.net, Rick@ShellZ.nl
Get your free ShellZ at www.shellz.nl now!
******************************************



To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message


From owner-freebsd-smp  Sat Dec  9 17:16:52 2000
From owner-freebsd-smp@FreeBSD.ORG  Sat Dec  9 17:16:50 2000
Return-Path: Delivered-To: freebsd-smp@freebsd.org
Received: from sharmas.dhs.org (c62443-a.frmt1.sfba.home.com [24.0.69.165])
	by hub.freebsd.org (Postfix) with ESMTP id 33A3E37B400
	for; Sat,  9 Dec 2000 17:16:50 -0800 (PST)
Received: from sharmas.dhs.org ([192.168.1.17])
	by sharmas.dhs.org (8.9.3/8.9.3) with ESMTP id RAA29155;
	Sat, 9 Dec 2000 17:15:08 -0800
Message-ID: <3A318757.4040108@sharmas.dhs.org>
Date: Fri, 08 Dec 2000 17:13:59 -0800
From: Arun Sharma User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; m18) Gecko/20001121
X-Accept-Language: en
MIME-Version: 1.0
To: Chuck Paterson Cc: smp@FreeBSD.ORG
Subject: Re: Userland atomic assignments
References: <200012091602.eB9G2UH06249@berserker.bsdi.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Sender: owner-freebsd-smp@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.org

Chuck Paterson wrote:

> }
> }However, the complexity comes from memory ordering issues. All of x86 boxes
> }ensure sequential consistency.
> }
> 
> Sequenctial consistency from a single processor, which may well
> be what you meant.

Actually, what I wrote in my previous mail wasn't completely accurate 
about sequential
consistency. (c == d) => (a == b) basically means writes happen in program
order as you say above.

> Writes from different processors to different cache lines are not
> ordered with respect to one another.  Also writes from one processor
> are not at all ordered with respect to reads from another processor.
> 
> What is guaranteed is that writes from a single processor will be
> ordered with respect to one another as viewed from any processor.


What sequential consistency guarantees is that writes as seen from
some other processor would make sense for _some_ arbitrary intermixing
of writes from other processors.   

A more formal definition:

http://whatis.techtarget.com/WhatIs_Definition_Page/0,4152,212962,00.html
   -Arun





To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message


From owner-freebsd-smp  Sat Dec  9 20:30:56 2000
From owner-freebsd-smp@FreeBSD.ORG  Sat Dec  9 20:30:54 2000
Return-Path: Delivered-To: freebsd-smp@freebsd.org
Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101])
	by hub.freebsd.org (Postfix) with ESMTP id 4E6C537B400
	for; Sat,  9 Dec 2000 20:30:54 -0800 (PST)
Received: (from dan@localhost)
	by dan.emsphone.com (8.11.1/8.11.1) id eBA4Ujg21368;
	Sat, 9 Dec 2000 22:30:45 -0600 (CST)
	(envelope-from dan)
Date: Sat, 9 Dec 2000 22:30:45 -0600
From: Dan Nelson To: Rick Jansen Cc: smp@FreeBSD.ORG
Subject: Re: MySQLd using just one CPU
Message-ID: <20001209223045.A541@dan.emsphone.com>
References: Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.3.12i
In-Reply-To:; from "Rick Jansen" on Sun Dec 10 01:37:12 GMT 2000
X-OS: FreeBSD 5.0-CURRENT
Sender: dan@dan.emsphone.com
Sender: owner-freebsd-smp@FreeBSD.ORG
Precedence: bulk
X-Loop: FreeBSD.org

In the last episode (Dec 10), Rick Jansen said:
> I have a problem with my dedicated MySQL server running on
> 4.2-STABLE. When i stress the daemon to the max, and i look at top,
> the WCPU and CPU fields are both around 99%. Which is what i want.
> BUT... Looking at the CPU States, it says the machine is around 49%
> idle! This leads me to conclude that MySQL is using just one
> processor. In Linux, there are a lot of mysqld processes because of
> the different threadpackage (LinuxThreads instead of MIT). Is there a
> threadpackage of this kind for FreeBSD aswell? I guess that would use
> both cpu's..

Try ports/devel/linuxthreads.  You'll have to figure out how to get
mysql to use it on your own :)

-- 
	Dan Nelson
	dnelson@emsphone.com


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-smp" in the body of the message