From owner-freebsd-cluster@FreeBSD.ORG Sun Apr 4 03:33:46 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 964CD16A4CE for ; Sun, 4 Apr 2004 03:33:46 -0700 (PDT) Received: from web41501.mail.yahoo.com (web41501.mail.yahoo.com [66.218.93.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 6ECF443D41 for ; Sun, 4 Apr 2004 03:33:46 -0700 (PDT) (envelope-from asporner@yahoo.com) Message-ID: <20040404103346.76151.qmail@web41501.mail.yahoo.com> Received: from [80.131.177.252] by web41501.mail.yahoo.com via HTTP; Sun, 04 Apr 2004 03:33:46 PDT Date: Sun, 4 Apr 2004 03:33:46 -0700 (PDT) From: Andy Sporner To: Michael McDonald , freebsd-cluster@freebsd.org In-Reply-To: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Re: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Apr 2004 10:33:46 -0000 Hi Michael, I am going to presume you mean a compute cluster (there are other types) based on the context of your message. I work for a company that makes network switches and we have seen best case where a server that is doing for instance a firewall application can do a maximum of about 30% of a gigabit. As this is more network centric rather than compute centric, I would expect that your utilitization would be lower than this. For 4 nodes I would think a standard gigabit switch would be enough. Managed allows you to have vlans, which doesn't seem to be requisite in your application. Additionally, haveing a separate network for the other traffic seems more trouble than it's worth. If you want reliability, and can find a bonding ethernet driver, than I would suggest creating a Adapter fault tolerance virtual adaptor and then use 2 gigabit adapters on the server and send them through two separate switches. I have looked in the last months for such a driver on Freebsd without luck. There was in times past an FEC driver. But all reports show that it is unstable. I have tried it anyways and can confirm that it is in fact.. unstable. There is a one to many configuration in netgraph that might however be usable. As for the software, somebody else will have to speak up. I only work on HA (and later Mosix) type clustering for network application scalablity, rather than compute farms. I would suggest if Ronald Milch is still listening that he would be the perfect answer man for your needs as this is his strong area. Best wishes for your project... Andy __________________________________ Do you Yahoo!? Yahoo! Small Business $15K Web Design Giveaway http://promotions.yahoo.com/design_giveaway/ From owner-freebsd-cluster@FreeBSD.ORG Sun Apr 4 03:39:26 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C41C216A4CE for ; Sun, 4 Apr 2004 03:39:26 -0700 (PDT) Received: from iota.root-servers.ch (iota.root-servers.ch [193.41.193.195]) by mx1.FreeBSD.org (Postfix) with SMTP id B54F143D1F for ; Sun, 4 Apr 2004 03:39:23 -0700 (PDT) (envelope-from gabriel_ambuehl@buz.ch) Received: (qmail 56952 invoked from network); 4 Apr 2004 10:39:14 -0000 Received: from 217-162-135-163.dclient.hispeed.ch (HELO ga) (217.162.135.163) by 0 with SMTP; 4 Apr 2004 10:39:14 -0000 Date: Sun, 4 Apr 2004 12:43:12 +0200 From: Gabriel Ambuehl Organization: BUZ Internet Services X-Priority: 3 (Normal) Message-ID: <987274454.20040404124312@buz.ch> To: Andy Sporner In-Reply-To: <20040404103346.76151.qmail@web41501.mail.yahoo.com> References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> <20040404103346.76151.qmail@web41501.mail.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: freebsd-cluster@freebsd.org Subject: Re[2]: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: gabriel_ambuehl@buz.ch List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Apr 2004 10:39:26 -0000 Sunday, April 4, 2004, 12:33:46 PM, you wrote: > As for the software, somebody else will have to > speak up. I only work on HA (and later Mosix) type > clustering for network application scalablity, > rather than compute farms. IIRC the DragonflyBSD guys are working on Single System Image clusters. But I doubt they are even anywhere near alpha stage at this point. Oh and Andy, how about that frep thing of yours? Is it going anywhere? Best regards, Gabriel From owner-freebsd-cluster@FreeBSD.ORG Sun Apr 4 04:31:21 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DB76716A4D0 for ; Sun, 4 Apr 2004 04:31:20 -0700 (PDT) Received: from ipx20050.ipxserver.de (ipx20050.ipxserver.de [80.190.243.49]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4FE7B43D54 for ; Sun, 4 Apr 2004 04:31:20 -0700 (PDT) (envelope-from andy@ipx20050.ipxserver.de) Received: from ipx20050.ipxserver.de (localhost.ipxserver.de [IPv6:::1]) by ipx20050.ipxserver.de (8.12.9/8.12.9) with ESMTP id i34BVBCi002935 for ; Sun, 4 Apr 2004 13:31:11 +0200 (CEST) Received: (from andy@localhost) by ipx20050.ipxserver.de (8.12.9/8.12.9/Submit) id i34BUtAh004422 for freebsd-cluster@freebsd.org; Sun, 4 Apr 2004 13:30:55 +0200 (CEST) Date: Sun, 4 Apr 2004 13:30:55 +0200 From: Andreas Schweitzer To: freebsd-cluster@freebsd.org Message-ID: <20040404113055.GA2677@ipx20050.ipxserver.de> References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> User-Agent: Mutt/1.4.1i Subject: Re: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Apr 2004 11:31:21 -0000 On Sat, Apr 03, 2004 at 08:52:02PM -0500, Michael McDonald wrote: First of all, it so depends what you want to do with your cluster. It seems you want to perform calculations. Do you have existing codes ? Ask yourself what they need. > I'm looking for recommendations at any level: > hardware: > managed/unmanaged switch? > specific brands/models? You mentioned MPI further down. If you have lots of communication, go for the fastets you can afford, otherwise it wil be the bottleneck. If you have lots of local calculations and an occasional network acces, eveen 100Mbit works. > diskless nodes? > network attached storage > processor speed vs. memory capacity vs. > comm. speed All depends on what you want to do with the cluster If you use NAS, ask yourself how often it will be accessed and how critical the access speed is. Local storage is a big plus if the calculations require static data which can be made available locally or if the calculations require disk storage themselves. And again, even gigabit network will be the bottleneck with todays CPU/disk/memory speeds. > software: > job submission/control We use OpenPBS, but it's quite out of date and requires a bit of hacking to make it run under FreeBSD 5.X. Also Sun Grid Engine seems to work. > node autoconfiguration I use clusterit (in the ports), but it's not "auto" - just convenient configuration - especially for a cluster with only a few nodes. > light, efficient comm. protocols > lam vs. mpich (currently using lam) We use LAM. I would say : test both. Cheers, Andreas Schweitzer From owner-freebsd-cluster@FreeBSD.ORG Sun Apr 4 16:30:01 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0B8A516A4CE for ; Sun, 4 Apr 2004 16:30:01 -0700 (PDT) Received: from catapult.dreamscape.com (catapult.dreamscape.com [206.64.128.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id AB2CD43D2D for ; Sun, 4 Apr 2004 16:30:00 -0700 (PDT) (envelope-from m.mcdonald@computer.org) Received: from mail2.dreamscape.com (mail2.dreamscape.com [206.64.128.18]) i34NTxfJ010779 for ; Sun, 4 Apr 2004 19:30:00 -0400 (EDT) Received: from MICHAELIWZHLNY (pool-141-149-206-40.syr.east.verizon.net [141.149.206.40])i34NTwwS009745 for ; Sun, 4 Apr 2004 19:29:59 -0400 (EDT) Message-ID: <002c01c41aa5$1fbd6b50$2f01a8c0@MICHAELIWZHLNY> From: "Michael McDonald" To: References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> <20040404113055.GA2677@ipx20050.ipxserver.de> Date: Sun, 4 Apr 2004 20:29:00 -0400 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 Subject: Re: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Michael McDonald List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Apr 2004 23:30:01 -0000 Thanks for the reponses. I should have made it clearer what my applications will be but you all doped it out pretty well. I'm setting it up for applications in computational chemistry and physics. A lot of eigenvalue and fourier computations. The codes use the netlib LAPACK and blas libraries and FFTW comes up a lot. For the most part, my codes have been data parallel and have invlolved broadcasting parameters and merging results at the end of distributed serial computations. They have involved many evaluations of small matrices and data sets and I get better speedups with simultaneous serial executions. Much of the work I'm looking towards would make use of a grid approach in the style of seti@home or the factoring projects. I would expect bursts of communication separated by periods of computation; Overall, communication wouldn't be so much of a bottleneck, but I'd like it to be fast when it does occur. If Andy Sporner's 30% figure holds up, I think local disks as buffers would allow the network access to be smeared out. I've got 40% in mind as an upper performance limit for ethernet due to collisions, but I can't back that up. Local buffers seem to allow for scheduling comm. so as to avoid collisions. Starting out with 100Mbit may make sense as a cheaper/simpler startup - no NIC purchases. After running some simulation and benchmarking with the apps, upgrading to Gigabit wouldn't be an undue burden. Some initial attempts at estimating communication & compute demands would be in order. Aside from the fiber options, is the cabling the same for 100Mbit and Gbit? Thanks to all, Mike McDonald From owner-freebsd-cluster@FreeBSD.ORG Mon Apr 5 01:24:28 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3042F16A4CE for ; Mon, 5 Apr 2004 01:24:28 -0700 (PDT) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.185]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5C29243D4C for ; Mon, 5 Apr 2004 01:24:27 -0700 (PDT) (envelope-from sporner@nentec.de) Received: from [212.227.126.161] (helo=mrelayng.kundenserver.de) by moutng.kundenserver.de with esmtp (Exim 3.35 #1) id 1BAPP7-0000YK-00; Mon, 05 Apr 2004 10:24:25 +0200 Received: from [194.25.215.66] (helo=gate.nentec.de) (TLSv1:EDH-RSA-DES-CBC3-SHA:168) (Exim 3.35 #1) id 1BAPP6-0001je-00; Mon, 05 Apr 2004 10:24:25 +0200 Received: from nenny.nentec.de (nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/) with ESMTP id i358OMq26791; Mon, 5 Apr 2004 10:24:22 +0200 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id i358OKN23221; Mon, 5 Apr 2004 10:24:21 +0200 Message-ID: <407117B4.2000800@nentec.de> Date: Mon, 05 Apr 2004 10:24:20 +0200 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Michael McDonald References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> <20040404113055.GA2677@ipx20050.ipxserver.de> <002c01c41aa5$1fbd6b50$2f01a8c0@MICHAELIWZHLNY> Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) X-Provags-ID: kundenserver.de abuse@kundenserver.de auth:56ea142331898a06f3703ddc80e12bc5 cc: freebsd-cluster@freebsd.org Subject: Re: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Apr 2004 08:24:28 -0000 Hi Michael, >For the most part, my codes have been data >parallel and have invlolved broadcasting >parameters and merging results at the end of >distributed serial computations. They have >involved many evaluations of small matrices >and data sets and I get better speedups with >simultaneous serial executions. Much of the >work I'm looking towards would make use >of a grid approach in the style of seti@home >or the factoring projects. > > Seti@home is very nice indeed. I am a participant as well (Captain Blank) >I would expect bursts of communication >separated by periods of computation; >Overall, communication wouldn't be >so much of a bottleneck, but I'd like it >to be fast when it does occur. If Andy >Sporner's 30% figure holds up, I think >local disks as buffers would allow the >network access to be smeared out. I've got >40% in mind as an upper performance limit >for ethernet due to collisions, but I can't back >that up. Local buffers seem to allow for >scheduling comm. so as to avoid collisions. > With a switch you wouldn't normally have enough collisions to worry about, unless you approach the bandwidth of the media. When I spoke of the 30% I was refering to a extended burst. We benchmark our firewall loadbalancer(nitro) with 4 firewalls and a traffic generation farm for entire weekends for reliability testing, this is where I observed this value. The Firewall machines are 2.8 Ghz Athlon servers. > >Starting out with 100Mbit may make sense as >a cheaper/simpler startup - no NIC >purchases. After running some simulation >and benchmarking with the apps, upgrading to Gigabit wouldn't be an undue burden. > That's not a bad idea, but many of the 1U servers (not to mention the better motherboards (such as ASUS) are with GB as standard). The switch is a little more expensive, but not that drastic. >Some initial attempts at estimating >communication & compute demands would >be in order. Aside from the fiber options, is the cabling the same for 100Mbit and Gbit? > Using Cat6 -- yes, but we have also used cat 5 with some success. If you start wth CAT-6 you will always win. Good luck! Andy From owner-freebsd-cluster@FreeBSD.ORG Mon Apr 5 01:37:19 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 001AA16A4CE for ; Mon, 5 Apr 2004 01:37:18 -0700 (PDT) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.185]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9508343D48 for ; Mon, 5 Apr 2004 01:37:18 -0700 (PDT) (envelope-from sporner@nentec.de) Received: from [212.227.126.162] (helo=mrelayng.kundenserver.de) by moutng.kundenserver.de with esmtp (Exim 3.35 #1) id 1BAPbZ-0003tw-00; Mon, 05 Apr 2004 10:37:17 +0200 Received: from [194.25.215.66] (helo=gate.nentec.de) (TLSv1:EDH-RSA-DES-CBC3-SHA:168) (Exim 3.35 #1) id 1BAPbZ-0001sN-00; Mon, 05 Apr 2004 10:37:17 +0200 Received: from nenny.nentec.de (nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/) with ESMTP id i358bEq28955; Mon, 5 Apr 2004 10:37:14 +0200 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id i358bDN24274; Mon, 5 Apr 2004 10:37:13 +0200 Message-ID: <40711AB9.6010602@nentec.de> Date: Mon, 05 Apr 2004 10:37:13 +0200 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: gabriel_ambuehl@buz.ch References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> <20040404103346.76151.qmail@web41501.mail.yahoo.com> <987274454.20040404124312@buz.ch> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) X-Provags-ID: kundenserver.de abuse@kundenserver.de auth:56ea142331898a06f3703ddc80e12bc5 cc: freebsd-cluster@freebsd.org Subject: Re: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Apr 2004 08:37:19 -0000 Gabriel Ambuehl wrote: >Sunday, April 4, 2004, 12:33:46 PM, you wrote: > > > >>As for the software, somebody else will have to >>speak up. I only work on HA (and later Mosix) type >>clustering for network application scalablity, >>rather than compute farms. >> >> > >IIRC the DragonflyBSD guys are working on Single System Image >clusters. But I doubt they are even anywhere near alpha stage at this >point. > >Oh and Andy, how about that frep thing of yours? Is it going anywhere? > > This is a news point. I had a rather nice suggestion from somebody "down under" that made sense and I wanted to get it in. At 20:00 (Berlin Standard time) I will release an initial release of FREP 2.0 Somebody pointed out that It was too bad I couldn't use 'triggers' to driver the replication. I realized that I had sort of that kind of thing already, but not generic enough for ordinary use outside of FREP. So now it is. For example, Whenever you modify /etc/inetd.conf the deaemon is sent a SIGHUP. Or if a file is created in a directory (such as a spool directory) that a program is called with the name of the file as an argument. This extended configuration also includes the original FREP stuff as an modular option. One can replace this module with a journal capture module as well. The modularity allows people to use the event triggers to be used without having to use FREP. FREP is then just an option as well as cluster locking of files (which will take a little longer--but the hooks are there). Sometime after this release there will be the actual tested FREP module as well to do the file replication. A "work in progress" version will be included for tonight. One of the main delays was that the kern_getcwd() functionality in 5.0 (and 5.2) is wholly unreliable. I had to incorporate a separate version to make it work correctly. Normally I hate such hacks, but it seems that the vfs_cache method that was used was not appearing to be 100% accurate (at least in the kernel). Many times when I would try to get a full filename I got errors (path component not a directory) when I passed a vnode of a file in to get the path name. In a file replication scheme--this is not acceptable. From owner-freebsd-cluster@FreeBSD.ORG Mon Apr 5 04:52:03 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CCC0316A4CE for ; Mon, 5 Apr 2004 04:52:03 -0700 (PDT) Received: from mail005.syd.optusnet.com.au (mail005.syd.optusnet.com.au [211.29.132.54]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8204243D1F for ; Mon, 5 Apr 2004 04:52:01 -0700 (PDT) (envelope-from Peter.Ross@alumni.tu-berlin.de) Received: from guckloch.zuhause (winax24-048.dialup.optusnet.com.au [211.29.117.48])i35Bpiu21444; Mon, 5 Apr 2004 21:51:45 +1000 Received: from guckloch.zuhause (localhost.zuhause [127.0.0.1]) by guckloch.zuhause (8.12.9/8.12.9) with ESMTP id i35BpAse000931; Mon, 5 Apr 2004 21:51:10 +1000 (EST) (envelope-from Peter.Ross@alumni.tu-berlin.de) Received: from localhost (petros@localhost)i35Bp6XP000928; Mon, 5 Apr 2004 21:51:07 +1000 (EST) X-Authentication-Warning: guckloch.zuhause: petros owned process doing -bs Date: Mon, 5 Apr 2004 21:51:06 +1000 (EST) From: Peter Ross X-X-Sender: petros@guckloch.zuhause To: Andy Sporner In-Reply-To: <40711AB9.6010602@nentec.de> Message-ID: <20040405211409.F644@guckloch.zuhause> References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> <987274454.20040404124312@buz.ch> <40711AB9.6010602@nentec.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-cluster@freebsd.org Subject: Re: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Apr 2004 11:52:03 -0000 On Mon, 5 Apr 2004, Andy Sporner wrote: > I had a rather nice suggestion from somebody "down under" that made > sense and I wanted to get it in. Propably it was me, a German in Melbourne;-) > At 20:00 (Berlin Standard time) I will release an initial release of > FREP 2.0 > > Somebody pointed out that It was too bad I couldn't use 'triggers' > to driver the replication. I realized that I had sort of that kind of > thing already, but not generic enough for ordinary use outside of > FREP. So now it is. Great. Congratulations! (I am a little bit ashamed. After a while of unemployment I started to think about some interesting things and discussed it, and when it began to get the right shape, I've got a job and became busy and if spare time was looming I became sick. But I hope it gets better..) > One of the main delays was that the kern_getcwd() functionality > in 5.0 (and 5.2) is wholly unreliable. I had to incorporate a separate > version to make it work correctly. Normally I hate such hacks, but > it seems that the vfs_cache method that was used was not appearing > to be 100% accurate (at least in the kernel). Many times when I > would try to get a full filename I got errors (path component not > a directory) when I passed a vnode of a file in to get the path name. > In a file replication scheme--this is not acceptable. I'm curious to see.. It's in the vnode layer so I assume it's filesystem independent? Regards Peter From owner-freebsd-cluster@FreeBSD.ORG Mon Apr 5 05:13:12 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 18AE716A4CE for ; Mon, 5 Apr 2004 05:13:12 -0700 (PDT) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.177]) by mx1.FreeBSD.org (Postfix) with ESMTP id 71EF743D67 for ; Mon, 5 Apr 2004 05:13:11 -0700 (PDT) (envelope-from sporner@nentec.de) Received: from [212.227.126.162] (helo=mrelayng.kundenserver.de) by moutng.kundenserver.de with esmtp (Exim 3.35 #1) id 1BASyQ-0001si-00; Mon, 05 Apr 2004 14:13:06 +0200 Received: from [194.25.215.66] (helo=gate.nentec.de) (TLSv1:EDH-RSA-DES-CBC3-SHA:168) (Exim 3.35 #1) id 1BASyQ-000586-00; Mon, 05 Apr 2004 14:13:06 +0200 Received: from nenny.nentec.de (nenny.nentec.de [153.92.64.1]) by gate.nentec.de (8.11.3/) with ESMTP id i35CD3q13683; Mon, 5 Apr 2004 14:13:03 +0200 Received: from nentec.de (andromeda.nentec.de [153.92.64.34]) by nenny.nentec.de (8.11.3/8.11.3) with ESMTP id i35CD0N17021; Mon, 5 Apr 2004 14:13:00 +0200 Message-ID: <40714D4C.5060207@nentec.de> Date: Mon, 05 Apr 2004 14:13:00 +0200 From: Andy Sporner User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2a) Gecko/20020910 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Peter Ross References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> <20040404103346.76151.qmail@web41501.mail.yahoo.com> <987274454.20040404124312@buz.ch> <40711AB9.6010602@nentec.de> <20040405211409.F644@guckloch.zuhause> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by AMaViS-perl11-milter (http://amavis.org/) X-Provags-ID: kundenserver.de abuse@kundenserver.de auth:56ea142331898a06f3703ddc80e12bc5 cc: freebsd-cluster@freebsd.org Subject: Re: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Apr 2004 12:13:12 -0000 Hi Peter, >Propably it was me, a German in Melbourne;-) > > I didn't want to cause any perhaps "undue" attention, but since you spoke up ;-) Yes it was your suggestions that I am referring to. >>One of the main delays was that the kern_getcwd() functionality >>in 5.0 (and 5.2) is wholly unreliable. I had to incorporate a separate >>version to make it work correctly. Normally I hate such hacks, but >>it seems that the vfs_cache method that was used was not appearing >>to be 100% accurate (at least in the kernel). Many times when I >>would try to get a full filename I got errors (path component not >>a directory) when I passed a vnode of a file in to get the path name. >>In a file replication scheme--this is not acceptable. >> >> > >I'm curious to see.. It's in the vnode layer so I assume it's filesystem >independent? > > That is correct. Gabriel suggested that somebody might do some sort of journaling thing with it. Now it is easier. Andy From owner-freebsd-cluster@FreeBSD.ORG Mon Apr 5 08:06:05 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DDC9416A4CF for ; Mon, 5 Apr 2004 08:06:05 -0700 (PDT) Received: from iota.root-servers.ch (iota.root-servers.ch [193.41.193.195]) by mx1.FreeBSD.org (Postfix) with SMTP id C847243D45 for ; Mon, 5 Apr 2004 08:06:02 -0700 (PDT) (envelope-from gabriel_ambuehl@buz.ch) Received: (qmail 75671 invoked from network); 5 Apr 2004 15:06:01 -0000 Received: from 217-162-135-163.dclient.hispeed.ch (HELO ga) (217.162.135.163) by 0 with SMTP; 5 Apr 2004 15:06:01 -0000 Date: Mon, 5 Apr 2004 17:09:49 +0200 From: Gabriel Ambuehl Organization: BUZ Internet Services X-Priority: 3 (Normal) Message-ID: <1131541187.20040405170949@buz.ch> To: Andy Sporner In-Reply-To: <40711AB9.6010602@nentec.de> References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> <20040404103346.76151.qmail@web41501.mail.yahoo.com> <987274454.20040404124312@buz.ch> <40711AB9.6010602@nentec.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: freebsd-cluster@freebsd.org Subject: Re[2]: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: gabriel_ambuehl@buz.ch List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Apr 2004 15:06:06 -0000 Hello Andy, Monday, April 5, 2004, 10:37:13 AM, you wrote: > At 20:00 (Berlin Standard time) I will release an initial release of > FREP 2.0 > Somebody pointed out that It was too bad I couldn't use 'triggers' > to driver the replication. I realized that I had sort of that kind of > thing already, but not generic enough for ordinary use outside of > FREP. So now it is. > For example, Whenever you modify /etc/inetd.conf the deaemon is > sent a SIGHUP. Or if a file is created in a directory (such as a spool > directory) that a program is called with the name of the file as an > argument. You might want to submit that to the main tree once it becomes stable, it would be immensely useful for on access anti virus scans, for one and I'm sure there are other areas that currently don't come to my mind... > to be 100% accurate (at least in the kernel). Many times when I > would try to get a full filename I got errors (path component not > a directory) when I passed a vnode of a file in to get the path name. > In a file replication scheme--this is not acceptable. Great. I'll surely beat on it over Easter holidays (I doubt I'll have much time before though). Best regards, Gabriel From owner-freebsd-cluster@FreeBSD.ORG Tue Apr 6 10:34:47 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 51C6716A4CE for ; Tue, 6 Apr 2004 10:34:47 -0700 (PDT) Received: from iota.root-servers.ch (iota.root-servers.ch [193.41.193.195]) by mx1.FreeBSD.org (Postfix) with SMTP id 0589A43D3F for ; Tue, 6 Apr 2004 10:34:46 -0700 (PDT) (envelope-from gabriel_ambuehl@buz.ch) Received: (qmail 5121 invoked from network); 6 Apr 2004 17:33:55 -0000 Received: from 217-162-135-163.dclient.hispeed.ch (HELO ga) (217.162.135.163) by 0 with SMTP; 6 Apr 2004 17:33:55 -0000 Date: Tue, 6 Apr 2004 19:37:24 +0200 From: Gabriel Ambuehl Organization: BUZ Internet Services X-Priority: 3 (Normal) Message-ID: <1545919554.20040406193724@buz.ch> To: Andy Sporner In-Reply-To: <407117B4.2000800@nentec.de> References: <002401c419e7$76692ac0$2f01a8c0@MICHAELIWZHLNY> <20040404113055.GA2677@ipx20050.ipxserver.de> <002c01c41aa5$1fbd6b50$2f01a8c0@MICHAELIWZHLNY> <407117B4.2000800@nentec.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: freebsd-cluster@freebsd.org Subject: Re[2]: Request for Cluster Recommendations X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: gabriel_ambuehl@buz.ch List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Apr 2004 17:34:47 -0000 Hello Andy, Monday, April 5, 2004, 10:24:20 AM, you wrote: > Seti@home is very nice indeed. I am a participant as > well (Captain Blank) Personally, I tend to think stuff like Folding is a bit more useful than Seti which just computes something (without much theory behind it to boot). >>Starting out with 100Mbit may make sense as >>a cheaper/simpler startup - no NIC >>purchases. After running some simulation >>and benchmarking with the apps, upgrading to Gigabit wouldn't be an undue burden. > That's not a bad idea, but many of the 1U servers (not to mention > the better motherboards (such as ASUS) are with GB as standard). > The switch is a little more expensive, but not that drastic. Even the cheaper are quickly shipping with GigE as standard. In fact, the next Southbridge revisions all incorporate GigE (also decoupled from PCI BUS, so you can actually do something besides pushing data over the LAN). >>Some initial attempts at estimating >>communication & compute demands would >>be in order. Aside from the fiber options, is the cabling the same for 100Mbit and Gbit? > Using Cat6 -- yes, but we have also used cat 5 with some success. > If you start wth CAT-6 you will always win. Did you release FREP? If so, were? Best regards, Gabriel From owner-freebsd-cluster@FreeBSD.ORG Sat Apr 10 15:21:27 2004 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6E7BD16A4CE; Sat, 10 Apr 2004 15:21:27 -0700 (PDT) Received: from ms-smtp-02.nyroc.rr.com (ms-smtp-02.nyroc.rr.com [24.24.2.56]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0097743D1D; Sat, 10 Apr 2004 15:21:27 -0700 (PDT) (envelope-from jracine@maxwell.syr.edu) Received: from [24.59.145.52] (syr-24-59-145-52.twcny.rr.com [24.59.145.52]) i3AMLOuV005168; Sat, 10 Apr 2004 18:21:25 -0400 (EDT) From: Jeffrey Racine To: freebsd-amd64@freebsd.org, freebsd-cluster@freebsd.org Content-Type: text/plain Organization: Syracuse University Message-Id: <1081635706.67575.26.camel@x1-6-00-b0-d0-c2-67-0e.twcny.rr.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.5.6.2FreeBSD GNOME Team Port Date: Sat, 10 Apr 2004 18:21:46 -0400 Content-Transfer-Encoding: 7bit X-Virus-Scanned: Symantec AntiVirus Scan Engine Subject: LAM MPI on dual processor opteron box sees only one cpu... X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Apr 2004 22:21:27 -0000 Hi. I am converging on getting a new dual opteron box running. Now I am setting up and testing LAM MPI, however, the OS is not farming out the job as expected, and only sees one processor. This runs fine on RH 7.3 and RH 9.0 both on a cluster and on a dual processor PIV desktop. I am running 5-current. Basically, mpirun -np 1 binaryfile has the same runtime as mpirun -np 2 binaryfile, while on the dual PIV box it runs in half the time. When I check top, mpirun -np 2 both run on CPU 0... here is the relevant portion from top with -np 2... 9306 jracine 4 0 7188K 2448K sbwait 0 0:03 19.53% 19.53% n_lam 29307 jracine 119 0 7148K 2372K CPU0 0 0:03 19.53% 19.53% n_lam I include output from laminfo, dmesg (cpu relevnt info), and lamboot -d bhost.lam... any suggestions most appreciated, and thanks in advance! -- laminfo LAM/MPI: 7.0.4 Prefix: /usr/local Architecture: amd64-unknown-freebsd5.2 Configured by: root Configured on: Sat Apr 10 11:22:02 EDT 2004 Configure host: jracine.maxwell.syr.edu C bindings: yes C++ bindings: yes Fortran bindings: yes C profiling: yes C++ profiling: yes Fortran profiling: yes ROMIO support: yes IMPI support: no Debug support: no Purify clean: no SSI boot: globus (Module v0.5) SSI boot: rsh (Module v1.0) SSI coll: lam_basic (Module v7.0) SSI coll: smp (Module v1.0) SSI rpi: crtcp (Module v1.0.1) SSI rpi: lamd (Module v7.0) SSI rpi: sysv (Module v7.0) SSI rpi: tcp (Module v7.0) SSI rpi: usysv (Module v7.0) -- dmesg sees two cpus... CPU: AMD Opteron(tm) Processor 248 (2205.02-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0xf58 Stepping = 8 Features=0x78bfbff AMD Features=0xe0500800 real memory = 3623813120 (3455 MB) avail memory = 3494363136 (3332 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 -- bhost has the requisite information 128.230.130.10 cpu=2 user=jracine -- Here are the results from lamboot -d bhost.lam -bash-2.05b$ lamboot -d ~/bhost.lam n0<29283> ssi:boot: Opening n0<29283> ssi:boot: opening module globus n0<29283> ssi:boot: initializing module globus n0<29283> ssi:boot:globus: globus-job-run not found, globus boot will not run n0<29283> ssi:boot: module not available: globus n0<29283> ssi:boot: opening module rsh n0<29283> ssi:boot: initializing module rsh n0<29283> ssi:boot:rsh: module initializing n0<29283> ssi:boot:rsh:agent: rsh n0<29283> ssi:boot:rsh:username: n0<29283> ssi:boot:rsh:verbose: 1000 n0<29283> ssi:boot:rsh:algorithm: linear n0<29283> ssi:boot:rsh:priority: 10 n0<29283> ssi:boot: module available: rsh, priority: 10 n0<29283> ssi:boot: finalizing module globus n0<29283> ssi:boot:globus: finalizing n0<29283> ssi:boot: closing module globus n0<29283> ssi:boot: Selected boot module rsh LAM 7.0.4/MPI 2 C++/ROMIO - Indiana University n0<29283> ssi:boot:base: looking for boot schema in following directories: n0<29283> ssi:boot:base: n0<29283> ssi:boot:base: $TROLLIUSHOME/etc n0<29283> ssi:boot:base: $LAMHOME/etc n0<29283> ssi:boot:base: /usr/local/etc n0<29283> ssi:boot:base: looking for boot schema file: n0<29283> ssi:boot:base: /home/jracine/bhost.lam n0<29283> ssi:boot:base: found boot schema: /home/jracine/bhost.lam n0<29283> ssi:boot:rsh: found the following hosts: n0<29283> ssi:boot:rsh: n0 jracine.maxwell.syr.edu (cpu=2) n0<29283> ssi:boot:rsh: resolved hosts: n0<29283> ssi:boot:rsh: n0 jracine.maxwell.syr.edu --> 128.230.130.10 (origin)n0<29283> ssi:boot:rsh: starting RTE procs n0<29283> ssi:boot:base:linear: starting n0<29283> ssi:boot:base:server: opening server TCP socket n0<29283> ssi:boot:base:server: opened port 49832 n0<29283> ssi:boot:base:linear: booting n0 (jracine.maxwell.syr.edu) n0<29283> ssi:boot:rsh: starting lamd on (jracine.maxwell.syr.edu) n0<29283> ssi:boot:rsh: starting on n0 (jracine.maxwell.syr.edu): hboot -t -c lam-conf.lamd -d -I -H 128.230.130.10 -P 49832 -n 0 -o 0 n0<29283> ssi:boot:rsh: launching locally hboot: performing tkill hboot: tkill -d tkill: setting prefix to (null) tkill: setting suffix to (null) tkill: got killname back: /tmp/lam-jracine@jracine.maxwell.syr.edu/lam-killfile tkill: removing socket file ... tkill: socket file: /tmp/lam-jracine@jracine.maxwell.syr.edu/lam-kernel-socketd tkill: removing IO daemon socket file ... tkill: IO daemon socket file: /tmp/lam-jracine@jracine.maxwell.syr.edu/lam-io-socket tkill: f_kill = "/tmp/lam-jracine@jracine.maxwell.syr.edu/lam-killfile" tkill: nothing to kill: "/tmp/lam-jracine@jracine.maxwell.syr.edu/lam-killfile" hboot: booting... hboot: fork /usr/local/bin/lamd [1] 29286 lamd -H 128.230.130.10 -P 49832 -n 0 -o 0 -d n0<29283> ssi:boot:rsh: successfully launched on n0 (jracine.maxwell.syr.edu) n0<29283> ssi:boot:base:server: expecting connection from finite list hboot: attempting to execute n-1<29286> ssi:boot: Opening n-1<29286> ssi:boot: opening module globus n-1<29286> ssi:boot: initializing module globus n-1<29286> ssi:boot:globus: globus-job-run not found, globus boot will not run n-1<29286> ssi:boot: module not available: globus n-1<29286> ssi:boot: opening module rsh n-1<29286> ssi:boot: initializing module rsh n-1<29286> ssi:boot:rsh: module initializing n-1<29286> ssi:boot:rsh:agent: rsh n-1<29286> ssi:boot:rsh:username: n-1<29286> ssi:boot:rsh:verbose: 1000 n-1<29286> ssi:boot:rsh:algorithm: linear n-1<29286> ssi:boot:rsh:priority: 10 n-1<29286> ssi:boot: module available: rsh, priority: 10 n-1<29286> ssi:boot: finalizing module globus n-1<29286> ssi:boot:globus: finalizing n-1<29286> ssi:boot: closing module globus n-1<29286> ssi:boot: Selected boot module rsh n0<29283> ssi:boot:base:server: got connection from 128.230.130.10 n0<29283> ssi:boot:base:server: this connection is expected (n0) n0<29283> ssi:boot:base:server: remote lamd is at 128.230.130.10:50206 n0<29283> ssi:boot:base:server: closing server socket n0<29283> ssi:boot:base:server: connecting to lamd at 128.230.130.10:49833 n0<29283> ssi:boot:base:server: connected n0<29283> ssi:boot:base:server: sending number of links (1) n0<29283> ssi:boot:base:server: sending info: n0 (jracine.maxwell.syr.edu) n0<29283> ssi:boot:base:server: finished sending n0<29283> ssi:boot:base:server: disconnected from 128.230.130.10:49833 n0<29283> ssi:boot:base:linear: finished n0<29283> ssi:boot:rsh: all RTE procs started n0<29283> ssi:boot:rsh: finalizing n0<29283> ssi:boot: Closing n-1<29286> ssi:boot:rsh: finalizing n-1<29286> ssi:boot: Closing