From owner-freebsd-questions Mon Nov 26 3: 4:20 2001 Delivered-To: freebsd-questions@freebsd.org Received: from mpehp1.mpe-garching.mpg.de (mpehp1.mpe-garching.mpg.de [130.183.70.10]) by hub.freebsd.org (Postfix) with ESMTP id AB94D37B439 for ; Mon, 26 Nov 2001 03:04:08 -0800 (PST) Received: from robert2.mpe-garching.mpg.de (robert2.mpe-garching.mpg.de [130.183.136.59]) by mpehp1.mpe-garching.mpg.de (8.8.6 (PHNE_17135)/8.8.6) with ESMTP id MAA27175; Mon, 26 Nov 2001 12:04:06 +0100 (MET) Received: (from sutter@localhost) by robert2.mpe-garching.mpg.de (8.11.6/8.11.6) id fAQB4Vx29563; Mon, 26 Nov 2001 12:04:31 +0100 (CET) (envelope-from sutter) Date: Mon, 26 Nov 2001 12:04:31 +0100 From: Robert Suetterlin To: Ted Mittelstaedt Cc: freebsd-questions@freebsd.org Subject: storing 10TB in 494x435x248mm, with power of 200W (at 28VDC) (was: why are You asking here) Message-ID: <20011126120431.C1170@robert2.mpe-garching.mpg.de> References: <20011123175912.B1170@robert2.mpe-garching.mpg.de> <000001c1744d$ca806340$1401a8c0@tedm.placo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <000001c1744d$ca806340$1401a8c0@tedm.placo.com>; from tedm@toybox.placo.com on Fri, Nov 23, 2001 at 10:36:40AM -0800 Sender: owner-freebsd-questions@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Hello Ted, thanks for discussing this problem so far, it really helps making up my mind when I get different opinions on this topic. [...] > >And we must have a solution > >that can be serviced by astronauts / kosmonauts easily. > > > what hardware are they already using and how much data is it storing? > Maybe someone can write a FreeBSD driver for it. [...] for our current experiment on the ISS we use a venerable sun laptop that contains the control-software and records housekeeping data (like 1-5MB every month). The real data is just standard video images recorded by Standard VCRs. These are no options in the planned for experiment. [...] > >> 3) What exactly is a HSM going to do for you? > >HSM is useful if You have a hirarchy of storage with different > >qualities. > That's what I said. > > You indicated that your doing archival only - ie: "one quality" in your > terms. Is this not true? [...] Sorry. I seem to keep to much of my thoughts to myself: A data rate of 260MByte per second, the size and power limits seemed to prohibit real time compression. (We need 'lossless' compression, and there has not been an algorithm selected / designed yet.) I concluded we would have to record the 45 Minute 260MB/s databursts to some kind of fast intermediate storage (for example RAM or Solid State Disks, etc.). Then we could spend some time (e.g. a day) on transferring that data to permanent storage and possibly reprocessing / compressing offline. Sometimes experimentalists would like to read some 'old' data back into the machinery to do a little analysis online (via telescience) before a shuttle takes the 'tapes' back to earth. In addition the system should 'automatically' record metadata and implement redundancy. To me there seemed to exist two possible solutions (except for the completely handmade thingy): 1) a kind of batchmode system running {record, reprocess, store} and {reread, reprocess, manipulate} jobs that would be similar to a VCR and would require implementing metadata and redundancy 'by hand'. 2) or using some kind of storage manager, that could handle these tasks and manage resources 'automatically'. [...] > >150TB of data per year. If there is a > >shuttle mission each month, they will have to transport 10TB of data > >each time. > So then have someone design an optical solution for you. [...] I have monitored the 'optical solutions' market for three years now and I do not trust in them beeing able to deliver better data density than magnetic media in the next ten years... but then I would be delighted if they proved me wrong. [...] > >I also would like to use the > >same general technique today that we will launch in five years (and then > >use for up to ten more). So I would like to use standard hardware and > >software solutions. These will most likely adept to the future > >automatically. > Any standard solution will be hopelessly obsolete 10 years from now and > getting > parts for it will cost astronomically. [...] You are completely right --- talking about fixed hardware. (I see that my paragraph hints to such a thing.) Yet I thought about a combination of software and hardware, where both could be upgraded independently while still keeping standard interfaces available, and called that a 'standard solution'. I mean something like Intel CPU, PC Architecture and *BSD. All have changed over the last ten years quite a lot. But still I could run a software that would rely on 'standard' interfaces (like files, ports, pipes, etc.) from ten years back on todays most modern hardware and newest *BSD version. And the prices would even have dropped. [...] > > An HSM would be an important building block in my > >concept of a solution to the data storage problem, as it relieves me of > >all the problems connected with hirarchical storage management. > No, it relieves you of having to redesign a storage solution sometime in the > future when you START needing HSM. [...] Exactly that is what I was looking for. A solution that can be deployed now. That can already be used and experimented with at the same time that the experiment is build and improved. And it's components will evolve on market preasure over the next five years. [...] > I think that your grossly underestimating the technical problems of handling > 10TB of data a month in a 494x435x248mm space, with power of 200W (at 28VDC). [...] > You could end up with a need for 50TB of data [...] > You have an application here that's crying out for a custom-built > high-capacity data storage device that takes small space and little > power and your screwing around looking for an off the shelf HSM array > that will do it. Such an animal doesen't exist. You need to take > that R&D budget you have and put it towards R&D with one of the major > technical players - perhaps IBM - that can supplement your budget. Of > course your going to end up with a custom-built solution but that's > the entire point of the exercise as then such solution can be > translated into a real product. > > Instead of attempting to get some sort of standard now, what you need > to do is trust in the Capatalistic system to use whatever work your > R&D budget creates and make commercial products out of it. Then 10 > years from now such high-capacity arrays will cost $100 and be > available from Costco. Attempting to use off-the-shelf products now > does not help anybody realize any R&D leverage from your budget - > which is the entire point of the money spent on ISS. > > The dollars are available from the hardware manufacturers. Your > project would have high marketing value to them. Leverage that value! Yes. Maybe that is the best strategy to go with... and it would be in the best spirit of the ISS project. Considering my 'spare' time and all I heard in several discussions this will be a likely solution. Regards, Robert. -- Robert Suetterlin (robert@mpe.mpg.de) phone: (+49)89 / 30000-3546 fax: (+49)89 / 30000-3950 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message