From owner-freebsd-cluster@FreeBSD.ORG Tue Nov 8 17:21:45 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CC5F616A41F for ; Tue, 8 Nov 2005 17:21:45 +0000 (GMT) (envelope-from clusterbuilder@gmail.com) Received: from xproxy.gmail.com (xproxy.gmail.com [66.249.82.205]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2089643D45 for ; Tue, 8 Nov 2005 17:21:44 +0000 (GMT) (envelope-from clusterbuilder@gmail.com) Received: by xproxy.gmail.com with SMTP id s9so698784wxc for ; Tue, 08 Nov 2005 09:21:44 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:mime-version:content-type; b=i2J51hy2Li8miiTD9x150rJzSBVRy5dIGwbz+CoKWGOT5zg0AhvflNrPBefBTR4yBJO2hYO5QoJcJ4YtpHnT2hrwLfqUu0LgZIxEMyfdwlhVRjAMX0Dl0OamrwS3MCL401QdthSmZ5iH2KWudZmGrdAwKlxNqnSpCx8Fl86JRY0= Received: by 10.70.78.8 with SMTP id a8mr3063175wxb; Tue, 08 Nov 2005 09:21:44 -0800 (PST) Received: by 10.70.70.9 with HTTP; Tue, 8 Nov 2005 09:21:44 -0800 (PST) Message-ID: Date: Tue, 8 Nov 2005 10:21:44 -0700 From: Nick I To: freebsd-cluster@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: backbone of a large-file streaming system X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Nov 2005 17:21:45 -0000 Hi, I help maintain a Web site at www.clusterbuilder.org. You might have seen before that I have a section called Ask the Cluster Expert, where I am building a knowledgebase of cluster and grid information= . When someone asks a question I am researching the answer to build this knowledgebase out. I received the following question: *"I am looking for a variety of solutions to be a backbone of a large-file streaming system providing thousands of concurrent download streams. Preferably commodity hardware and Linux, though I'm open to commercial solutions."* I am wondering if anyone here has suggestions of what applications will wor= k best for this type of setup. You can respond to the question at www.clusterbuilder.org/FAQ or respond in email. Thanks, Nick From owner-freebsd-cluster@FreeBSD.ORG Tue Nov 8 19:33:37 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 39CFB16A421 for ; Tue, 8 Nov 2005 19:33:37 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 22BC143D45 for ; Tue, 8 Nov 2005 19:33:28 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id jA8JXRG1046080; Tue, 8 Nov 2005 13:33:27 -0600 (CST) (envelope-from anderson@centtech.com) Message-ID: <4370FD83.3000500@centtech.com> Date: Tue, 08 Nov 2005 13:33:23 -0600 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20051021 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Nick I References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1166/Mon Nov 7 13:01:45 2005 on mh2.centtech.com X-Virus-Status: Clean Cc: freebsd-cluster@freebsd.org Subject: Re: backbone of a large-file streaming system X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Nov 2005 19:33:37 -0000 Nick I wrote: > Hi, > I help maintain a Web site at > www.clusterbuilder.org. > You might have seen before that I have a section called Ask the Cluster > Expert, where I am building a knowledgebase of cluster and grid information. > When someone asks a question I am researching the answer to build this > knowledgebase out. I received the following question: > > *"I am looking for a variety of solutions to be a backbone of a large-file > streaming system providing thousands of concurrent download streams. > Preferably commodity hardware and Linux, though I'm open to commercial > solutions."* > > I am wondering if anyone here has suggestions of what applications will work > best for this type of setup. You can respond to the question at > www.clusterbuilder.org/FAQ > or respond in email. I suppose the answer depends on the protocol used for the streaming downloads. Assuming HTTP, apache on a FreeBSD server should handle this nicely. Example is ftp.cdrom.com. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From owner-freebsd-cluster@FreeBSD.ORG Wed Nov 9 12:37:33 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5BBEA16A41F for ; Wed, 9 Nov 2005 12:37:33 +0000 (GMT) (envelope-from aggelis@gmail.com) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.205]) by mx1.FreeBSD.org (Postfix) with ESMTP id AA6CA43D45 for ; Wed, 9 Nov 2005 12:37:32 +0000 (GMT) (envelope-from aggelis@gmail.com) Received: by wproxy.gmail.com with SMTP id i31so119851wra for ; Wed, 09 Nov 2005 04:37:32 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=klLWIGDDTkzBJrq92Rq3VyWhpFBzIhXFD14bR/zvA3y/jN4qnhIoEMro0LWhorCoBvxUmXlc00Trp9zcVS/T5qhih/j3trT7k/ZdFFTfPFBOHraaYfyJiYK+PqBR1VpKD3tTmUTtj8AKnhogwVKP824D+nBRzBY0yTeDlOPqiSA= Received: by 10.54.152.2 with SMTP id z2mr577394wrd; Wed, 09 Nov 2005 04:37:32 -0800 (PST) Received: by 10.54.101.5 with HTTP; Wed, 9 Nov 2005 04:37:31 -0800 (PST) Message-ID: Date: Wed, 9 Nov 2005 14:37:31 +0200 From: Aggelis Aggelis To: Nick I In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: Cc: freebsd-cluster@freebsd.org Subject: Re: backbone of a large-file streaming system X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Nov 2005 12:37:33 -0000 On 11/8/05, Nick I wrote: > Hi, > I help maintain a Web site at > www.clusterbuilder.org. > You might have seen before that I have a section called Ask the Cluster > Expert, where I am building a knowledgebase of cluster and grid informati= on. > When someone asks a question I am researching the answer to build this > knowledgebase out. I received the following question: > > *"I am looking for a variety of solutions to be a backbone of a large-fil= e > streaming system providing thousands of concurrent download streams. > Preferably commodity hardware and Linux, though I'm open to commercial > solutions."* > > I am wondering if anyone here has suggestions of what applications will w= ork > best for this type of setup. You can respond to the question at > www.clusterbuilder.org/FAQ > or respond in email. > > Thanks, > Nick > _______________________________________________ > freebsd-cluster@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-cluster > To unsubscribe, send any mail to "freebsd-cluster-unsubscribe@freebsd.org= " > it all depends on job. This is the moto of clustering. >From what i have read the solution is load balancing/redundancy cluster. Probably something like what is described in the article at http://www.countersiege.com/doc/pfsync-carp/ at the section "Something Bigger" "This larger configuration is in place at a large educational institution, providing load balancing and redundancy for a cluster of web servers:" As far as the hardware , that depends on the kind and magnitude of load that you expect. there are also devices like eg this http://www.coyotepoint.com/extreme.htm Also read the pdf for what these devices are offering http://www.coyotepoint.com/pdfs/extreme_05.pdf. Anyway the key point here is on load balancing and redundancy client | | internet | Load Balancer/Firewall | | . .......... | | | | | | | server1 server2 serverN