Date: Sat, 27 Oct 2007 16:01:36 +0200 From: Ivan Voras <ivoras@freebsd.org> To: freebsd-questions@freebsd.org Subject: Re: Storage cluster with FreeBSD Message-ID: <ffvg87$p4n$1@ger.gmane.org> In-Reply-To: <20071026224437.27b778e2@meijome.net> References: <20071026104209.3c0628ba@meijome.net> <20071026102559.A83146@wojtek.tensor.gdynia.pl> <20071026212158.5c990f7a@meijome.net> <20071026134905.L84884@wojtek.tensor.gdynia.pl> <20071026224437.27b778e2@meijome.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Norberto Meijome wrote: > indeed. Well, as I said in OP, similar to what Lustre offers. I don't know what Lustre has so my response might or might not be what you need... > Let's see : what I am after is a way to hook up a few computers ( say, 6) with > a few HD (say, 6 x 400 GB), and setup across the whole thing a RAID that > would let me : > - keep running with no loss of data in case of a disk or node failure (or > minimal in case of a node failure) > > - can be accessed by clients over standard network protocols (NFS, CIFS, etc) > > - the clients see volumes of x GB (maybe 1 of the total size, maybe 3 of > different sizes...doesn't matter) > > All this until hear seems doable with FBSD + GEOM architecture. Now the tricky > ones : May or may not. Do you want to have the drives attached to individual machines, then exported via ggate, then mounted on each machine and then RAID-ed together? This looks like it needs a large overhead in administration. If you have the drives attached to individual machines and then one machine that imports them all in some way (ggate) and exports them via NFS and CIFS, then you have a single point of failure. In truth, there appears to be only one volume manager that's close to being usable, and that's ZFS in 7. But there's nothing automatic in the way it can use the drives and (re) export them via NFS, etc. There is no distributed file systems for FreeBSD. > - scales linearly - if I add 90 storage hosts, the system doesnt get bogged > down with the management of the disks / stripe distributoin The only things you have with GEOM are low-level building blocks: RAID transformations and network devices. If you calculate for yourself that, (a silly example) if you create a RAID1 out of 90 hosts you can survive the data being explicitely sent to each of the hosts individually, then go ahead. There is no magic or smart behaviour involved anywhere. > - I can add new hosts (from 6 to 10) and the new storage is available for the > cluster. If I had LVM, I could simply add the new disks to the physical group, > and then grow the logical volume... I don't see how 7.0's recursive > partitioning can help me here.... Can you please explain ? Only ZFS can do that on 7. In truth, FreeBSD is really bad for storage works, and probably noone has ever done what you need (it's just not supported). The reasons are: - there are no distributed file systems for freebsd (meaning ones that have built-in support for operation on multiple nodes) - UFS panics on the slightest IO error and is not mountable by multiple hosts at the same time *at all*, even over gmirror - gmirror over ggate sort-of works but ggate network and IO errors will (at best) disconnect one of the drives and you'll need to manually reconnect them to the gmirror. After that you'll need to rebuild the whole drives (which is slow over the network). I.e. there's no automatic failover.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ffvg87$p4n$1>