From owner-freebsd-hardware@FreeBSD.ORG Wed Aug 4 11:34:36 2004 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5225E16A4CF for ; Wed, 4 Aug 2004 11:34:36 +0000 (GMT) Received: from house.arach.net.au (house1.arach.net.au [203.30.44.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 6331B43D53 for ; Wed, 4 Aug 2004 11:34:34 +0000 (GMT) (envelope-from kat-free@kaqelectronics.dyndns.org) Received: (qmail 1432 invoked from network); 4 Aug 2004 11:34:31 -0000 Received: from unknown (HELO webserver.kaqelectronics.dyndns.org) (202.89.174.74) by house1.arach.net.au with SMTP for ; 4 Aug 2004 11:34:31 -0000 Received: from kaqelectronics.dyndns.org (kat.kaqelectronics.dyndns.org [192.168.0.2])id i74BaqDB048509 for ; Wed, 4 Aug 2004 19:36:52 +0800 (WST) (envelope-from kat-free@kaqelectronics.dyndns.org) Message-ID: <4110C9C7.6080506@kaqelectronics.dyndns.org> Date: Wed, 04 Aug 2004 19:34:31 +0800 From: Kathy Quinlan User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-au, en, en-us MIME-Version: 1.0 To: freebsd-hardware@freebsd.org Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Subject: Big Problem X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Aug 2004 11:34:36 -0000 Hi Guys and Gals, First off I am not a troll, this is a serious email. I can not go into to many fine points as I am bound by an NDA. The problem: I need to hold a text file in ram, the text file in the forseable future could be up to 10TB in size. My Options: Design a computer (probably multiple AMD 64's) to handle 10TB of memory (+ a few extra Gb of ram for system overhead) and hold the file in one physical computer system. Build a server farm and have each server hold a portion eg 4GB each Server (250 servers (plus a few extra for system overhead) The reason the file needs to be in ram is that I need speed of search for paterns in the data (less than 1 second to pull out relevent chunks) I am sure I have missed some options, right now I am just kicking ideas around, the software will be based on FreeBSD with some major modifications to address the large amount of ram (probably set it up as a virtual drive with one file) Regards, Kat.