From owner-freebsd-cluster@FreeBSD.ORG Sun Feb 28 11:36:39 2010 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9EAD2106564A for ; Sun, 28 Feb 2010 11:36:39 +0000 (UTC) (envelope-from escholtz@argonsoft.de) Received: from coyote.quickmin.net (coyote.quickmin.net [217.14.112.24]) by mx1.freebsd.org (Postfix) with ESMTP id 0E55E8FC12 for ; Sun, 28 Feb 2010 11:36:38 +0000 (UTC) Received: (qmail 53787 invoked from network); 28 Feb 2010 12:09:56 +0100 Received: by simscan 1.4.0 ppid: 53784, pid: 53785, t: 0.0148s scanners: clamav: 0.95.2/m:49/d:8609 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=argonsoft.de; b=jX5ashBQDLvGhqhjtt5sjhZbX92icMCdiwFGvImckpy/SW5j9AfH6ZXv1Rzaq4wiif9c5gwJ4RSjgUz0oq/07EN/2jMYL5v7/jykyGU4L0ZWeeBWczNVzSIMCLw8FLRI0QZ2Fpdw64lMvrHIYWJZIxWXp0jPpnfWpKl3qm5utns= ; Received: from 190-170-20-212.dsl.globvill.net (HELO erik-scholtzs-macbook-pro-15.local) (00000150@212.20.170.190) by coyote.quickmin.net with SMTP; 28 Feb 2010 12:09:56 +0100 Message-ID: <4B8A4EFF.9050207@argonsoft.de> Date: Sun, 28 Feb 2010 12:09:51 +0100 From: "Erik Scholtz, ArgonSoft GmbH" User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: freebsd-cluster@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Cluster Filesystem on FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 11:36:39 -0000 Hi, I did some research the last two weeks on how to build a cluster filesystem on FreeBSD. Since neither GFS ( http://sources.redhat.com/cluster/gfs/ ) nor OCFS2 ( http://oss.oracle.com/projects/ocfs2/ ) are ported to FreeBSD, I took a look at the GlusterFS-project ( http://www.gluster.org/ ), which also does not compile on FreeBSD 7.x, as well as PVFS ( http://www.pvfs.org/ ). My solution at the moment is, to rsync all filesystems once a minute, which is rather to rare. So I tried to get a hook with KQueue to rsync the filesystems on data-change. Unfortunatly I could not find a working solution (had a try with IO::KQueue using perl). NFS isn't really a solution, since it is slow (even on 1GBit) and is a single point of failure: Clustering the NFS server with machines in stand-by, that do a fail-over on a crash of the master-NFS is unstable and makes a lot more costs to the project. I'm with FreeBSD since the 2.1 release and I really love this system for it's performance and stability. But in the last year I came very often to a point, where I had no solution on FreeBSD and several solutions for Linux (like SAS-support, ...). How do you guys solve this problem (of a shared filesystem with rw-option)? Any hints are welcome, since I'm getting very frustrated at the moment. Greetings, Erik -- My blog: http://blog.elitecoderz.net From owner-freebsd-cluster@FreeBSD.ORG Sun Feb 28 23:21:43 2010 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3A991065670 for ; Sun, 28 Feb 2010 23:21:43 +0000 (UTC) (envelope-from luke@digital-crocus.com) Received: from mail.digital-crocus.com (node2.digital-crocus.com [91.209.244.128]) by mx1.freebsd.org (Postfix) with ESMTP id 5C2BB8FC14 for ; Sun, 28 Feb 2010 23:21:43 +0000 (UTC) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=dkselector; d=digital-crocus.com; h=Received:Received:Subject:From:To:Cc:In-Reply-To:References:Content-Type:Date:Message-Id:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Spam-Score:X-Digital-Crocus-Maillimit:X-Authenticated-Sender:X-Complaints:X-Admin:X-Abuse; b=ogY1yE2PK36FkkJg0zjdH65zhztqH0Kp90D+oG5hGFHLEtElfFrsAa3j1Ldtlx4hJRjxlFfx5Y7SV+R5Uumy+FBRxxGKU4CIeXL2GOMZ6gwCs2GCzxOFsDHMtQiyW6Hy; Received: from luke by mail.digital-crocus.com with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Nls65-000E1V-5x for freebsd-cluster@freebsd.org; Sun, 28 Feb 2010 22:58:49 +0000 Received: from 127cr.net ([78.105.122.99] helo=[192.168.1.20]) by mail.digital-crocus.com with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Nls64-000E1D-Ap; Sun, 28 Feb 2010 22:58:49 +0000 From: Luke Marsden To: "Erik Scholtz, ArgonSoft GmbH" In-Reply-To: <4B8A4EFF.9050207@argonsoft.de> References: <4B8A4EFF.9050207@argonsoft.de> Content-Type: text/plain Date: Sun, 28 Feb 2010 23:00:07 +0000 Message-Id: <1267398007.16514.89.camel@behemoth> Mime-Version: 1.0 X-Mailer: Evolution 2.26.1 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.4 X-Digital-Crocus-Maillimit: done X-Authenticated-Sender: luke X-Complaints: abuse@digital-crocus.com X-Admin: admin@digital-crocus.com X-Abuse: abuse@digital-crocus.com (Please include full headers in abuse reports) Cc: freebsd-cluster@freebsd.org Subject: Re: Cluster Filesystem on FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 23:21:43 -0000 Hi Erik, The best way I've found to monitor for filesystem activity on FreeBSD is with the audit_control kernel option. Recompile your kernel if necessary and pop this into /etc/security/audit_control (owned by root:wheel and 0600 perms): dir:/var/audit flags:fc,fd,fw minfree:20 naflags:lo policy:cnt filesz:0 Then this Python script (easily ported to Perl if you wish) will do the job of printing a path to stdout whenever a file is modified (and also pruning the praudit logfiles in /var/audit which aren't desirable for this application - we couldn't find a nicer way to do this): import popen2, sys, os pipe = popen2.Popen4("/usr/sbin/praudit /dev/auditpipe") count = 0 while True: line = pipe.fromchild.readline() if line=='': break # so that we die when praudit does if 'header' in line: header=line if '/path-to-monitor/' not in line: # useful to avoid a flood continue if 'path' in line: count += 1 if count % 1000 == 0: # clear out audit files to save disk space os.system('for X in /var/audit/*; do echo "" > "$X"; done') if 'fcntl' not in header: if line.split(',')[0].strip() == 'path': print "MODIFIED: "+line.split(',')[1].strip() sys.stdout.flush() Hope you find this useful, it's one small part of a FreeBSD-based web cluster product I'm working on, select parts of which we plan to release under the BSD license in the next few months in order to hopefully attract a developer community of FreeBSD cluster hackers. If you make any improvements to the script I'd appreciate getting them, just send them in to luke@hybrid-logic.co.uk Regarding filesystem replication (and without giving too much away) I recommend the ZFS filesystem with its fast, atomic snapshots and zfs send / zfs recv to do the job of rsync much more efficiently and at the block level. You won't find multi-master read/write with this solution on its own though. It is more useful to implement "streaming backups". At some point I'm planning to combine this approach with NFS failover in order to make every filesystem writable on every machine at all times, although this is not a requirement for our application right now. Stay tuned to www.hybrid-logic.co.uk for a timeline for open source availability of more of our cluster technology. (That part of the site isn't ready just yet, but should be in a few weeks.) Best Regards, Luke Marsden Hybrid Logic Ltd. On Sun, 2010-02-28 at 12:09 +0100, Erik Scholtz, ArgonSoft GmbH wrote: > Hi, > > I did some research the last two weeks on how to build a cluster > filesystem on FreeBSD. > > Since neither GFS ( http://sources.redhat.com/cluster/gfs/ ) nor OCFS2 ( > http://oss.oracle.com/projects/ocfs2/ ) are ported to FreeBSD, I took a > look at the GlusterFS-project ( http://www.gluster.org/ ), which also > does not compile on FreeBSD 7.x, as well as PVFS ( http://www.pvfs.org/ ). > > My solution at the moment is, to rsync all filesystems once a minute, > which is rather to rare. So I tried to get a hook with KQueue to rsync > the filesystems on data-change. Unfortunatly I could not find a working > solution (had a try with IO::KQueue using perl). > > NFS isn't really a solution, since it is slow (even on 1GBit) and is a > single point of failure: Clustering the NFS server with machines in > stand-by, that do a fail-over on a crash of the master-NFS is unstable > and makes a lot more costs to the project. > > I'm with FreeBSD since the 2.1 release and I really love this system for > it's performance and stability. But in the last year I came very often > to a point, where I had no solution on FreeBSD and several solutions for > Linux (like SAS-support, ...). > > > How do you guys solve this problem (of a shared filesystem with > rw-option)? Any hints are welcome, since I'm getting very frustrated at > the moment. > > Greetings, > Erik > From owner-freebsd-cluster@FreeBSD.ORG Mon Mar 1 09:28:39 2010 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33EC51065674 for ; Mon, 1 Mar 2010 09:28:39 +0000 (UTC) (envelope-from mandrei05@gmail.com) Received: from mail-fx0-f223.google.com (mail-fx0-f223.google.com [209.85.220.223]) by mx1.freebsd.org (Postfix) with ESMTP id AE9378FC22 for ; Mon, 1 Mar 2010 09:28:38 +0000 (UTC) Received: by fxm23 with SMTP id 23so447559fxm.3 for ; Mon, 01 Mar 2010 01:28:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=g9ZlYwFHaAzrzrd0hy0oenXJgLZvmtm26arutAD17zM=; b=KVzzagLGMbf39jH2kdNekoM9hLL/qBryTndNSwxoRVPM4FTS87iXvOego09o4Wyraa rmmKHVgZ0Hx0CXPtttfsp8Y2dFO2R7viSoPBhB1C/WLJz3bVpH+98SLTSvPZ5rG9PwIB K478D/+U9xr2lyxVbrIaCDMdtG+NPjxB0AFb0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=DcDwfmmhfynZpK1+3apPCOZz76UEH6qoW38703qKLIPZfOIPAjgat5DbQYQKRyFDCB qWr5fKMYVkPzB57UG/iUsxN1ZMOOuFIjUKJJJGl1f49C7qB0gzJ2zMO3t5zWmp64Gyha bzWRbD0d5y49BozpfS0fPnlwTGaeK9vE7pqtE= MIME-Version: 1.0 Received: by 10.239.166.84 with SMTP id a20mr414470hbe.116.1267434045561; Mon, 01 Mar 2010 01:00:45 -0800 (PST) In-Reply-To: <20100228120015.1D03310656B0@hub.freebsd.org> References: <20100228120015.1D03310656B0@hub.freebsd.org> Date: Mon, 1 Mar 2010 10:00:45 +0100 Message-ID: From: Andrei Manescu To: freebsd-cluster@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: freebsd-cluster Digest, Vol 136, Issue 1 X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 09:28:39 -0000 Hello Erik, Why would ggate geom module not suit you ? (You haven't mentioned it, but since you've been using FreeBSD since 2.1 I'm sure you're familiar with it). 2010/2/28 > Send freebsd-cluster mailing list submissions to > freebsd-cluster@freebsd.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.freebsd.org/mailman/listinfo/freebsd-cluster > or, via email, send a message with subject or body 'help' to > freebsd-cluster-request@freebsd.org > > You can reach the person managing the list at > freebsd-cluster-owner@freebsd.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of freebsd-cluster digest..." > > > Today's Topics: > > 1. Cluster Filesystem on FreeBSD (Erik Scholtz, ArgonSoft GmbH) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 28 Feb 2010 12:09:51 +0100 > From: "Erik Scholtz, ArgonSoft GmbH" > Subject: Cluster Filesystem on FreeBSD > To: freebsd-cluster@freebsd.org > Message-ID: <4B8A4EFF.9050207@argonsoft.de> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi, > > I did some research the last two weeks on how to build a cluster > filesystem on FreeBSD. > > Since neither GFS ( http://sources.redhat.com/cluster/gfs/ ) nor OCFS2 ( > http://oss.oracle.com/projects/ocfs2/ ) are ported to FreeBSD, I took a > look at the GlusterFS-project ( http://www.gluster.org/ ), which also > does not compile on FreeBSD 7.x, as well as PVFS ( http://www.pvfs.org/ ). > > My solution at the moment is, to rsync all filesystems once a minute, > which is rather to rare. So I tried to get a hook with KQueue to rsync > the filesystems on data-change. Unfortunatly I could not find a working > solution (had a try with IO::KQueue using perl). > > NFS isn't really a solution, since it is slow (even on 1GBit) and is a > single point of failure: Clustering the NFS server with machines in > stand-by, that do a fail-over on a crash of the master-NFS is unstable > and makes a lot more costs to the project. > > I'm with FreeBSD since the 2.1 release and I really love this system for > it's performance and stability. But in the last year I came very often > to a point, where I had no solution on FreeBSD and several solutions for > Linux (like SAS-support, ...). > > > How do you guys solve this problem (of a shared filesystem with > rw-option)? Any hints are welcome, since I'm getting very frustrated at > the moment. > > Greetings, > Erik > > -- > > My blog: http://blog.elitecoderz.net > > > ------------------------------ > > _______________________________________________ > freebsd-cluster@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-cluster > To unsubscribe, send any mail to "freebsd-cluster-unsubscribe@freebsd.org" > > > End of freebsd-cluster Digest, Vol 136, Issue 1 > *********************************************** > From owner-freebsd-cluster@FreeBSD.ORG Mon Mar 1 14:49:51 2010 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3967A106566B for ; Mon, 1 Mar 2010 14:49:51 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id E2B428FC12 for ; Mon, 1 Mar 2010 14:49:49 +0000 (UTC) Received: by gya1 with SMTP id 1so1414423gya.13 for ; Mon, 01 Mar 2010 06:49:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:to:cc :subject:in-reply-to:message-id:references:user-agent :x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; bh=rBS/J3TOxVsSqEpo924oXYEBcR2hThFLTKEJD2SPdvE=; b=bEnkdzOYOK08qhI4M7OSKLr14/cyRfrgdWxFwz/SscM8x+PIdAW2PKzpvAW3V4MHFJ nnrqeQ8bPugFLucKxWGdeB6PJj6xcTs6xN9eIvAMKWfa3Fvo8eTHcAOrcGsnUr9ZsFLO KSq30ILnuWbHuLaDMR4Ckh38T/ZjKwFpHqgVQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=lfY46pkIKPMQo25lHjLWxr1bLRbPc0Pz1qIJvkhZ/rJKJZMKxuR1EDszwpOLPpntAq Ob+vjaMlTAGjRDdEOYQ4PcOaZqD1v4vkCBtDMk9Fwpm58NkVsXQ7NBLHPp/3rHpAyWRi lUrehZISJLWcZqd+LwBG77lKj8VhEIDkuhdYM= Received: by 10.150.120.8 with SMTP id s8mr5203858ybc.127.1267453225796; Mon, 01 Mar 2010 06:20:25 -0800 (PST) Received: from centel.dataix.local (ppp-21.198.dialinfree.com [209.172.21.198]) by mx.google.com with ESMTPS id 7sm1195477ywf.55.2010.03.01.06.20.21 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 01 Mar 2010 06:20:24 -0800 (PST) Sender: "J. Hellenthal" Date: Mon, 1 Mar 2010 09:20:13 -0500 From: jhell To: Andrei Manescu In-Reply-To: Message-ID: References: <20100228120015.1D03310656B0@hub.freebsd.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-cluster@freebsd.org Subject: Re: freebsd-cluster Digest, Vol 136, Issue 1 X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 14:49:51 -0000 On Mon, 1 Mar 2010 04:00, mandrei05@ wrote: > Hello Erik, > > > Why would ggate geom module not suit you ? (You haven't mentioned it, but > since you've been using FreeBSD since 2.1 I'm sure you're familiar with it). > Can you explain how it would suit him ? "in a clustering situation". I am very interested to hear the details. >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of freebsd-cluster digest..." >> See above. Topic below. >> >> Today's Topics: >> >> 1. Cluster Filesystem on FreeBSD (Erik Scholtz, ArgonSoft GmbH) >> -- jhell From owner-freebsd-cluster@FreeBSD.ORG Mon Mar 1 18:39:21 2010 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E2671065675 for ; Mon, 1 Mar 2010 18:39:21 +0000 (UTC) (envelope-from salfrancl.listas@gmail.com) Received: from mail-bw0-f228.google.com (mail-bw0-f228.google.com [209.85.218.228]) by mx1.freebsd.org (Postfix) with ESMTP id E8B8C8FC26 for ; Mon, 1 Mar 2010 18:39:20 +0000 (UTC) Received: by bwz28 with SMTP id 28so672712bwz.14 for ; Mon, 01 Mar 2010 10:39:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=WxsRaJuMUzVfWvD/J7GwY4Z42IGVIcIpBchy7RC8l6k=; b=ElnZTIYQ/qkykAw9jafMDHswHxqq7frHj6GEUN7EnT69bm0aDRrXwbB99xtr5XEmWk 7rwRpMsnlQFBJgSLadxunxFrElr3hYo5FUnqF5H3rFXnOj5zh1LxGCEtk5TwaqnaLcVR M2TdPLX2QJk8Ue7Xmb/LYYcELsEGDKJBx2pVg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=IG+QaS7fOP9eozcRUchHnLtjBHHlqEKkUAy7leoQ8+PZBuS8ZgBUx7Y24nvOX9fxUq JZchqDyCqhEOZaqnudkP5hTiwL/ag357YsKpGIUi17DczCW2Xpjgr6nOe+POm/pnj40S O2ON7+4sGc/3MCBbA5Olp3gSNBXdxKFbkQRLQ= MIME-Version: 1.0 Received: by 10.204.33.196 with SMTP id i4mr3213526bkd.155.1267467110740; Mon, 01 Mar 2010 10:11:50 -0800 (PST) In-Reply-To: <4B8A4EFF.9050207@argonsoft.de> References: <4B8A4EFF.9050207@argonsoft.de> Date: Mon, 1 Mar 2010 13:11:50 -0500 Message-ID: From: Leinier Cruz Salfran To: freebsd-cluster@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: Cluster Filesystem on FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 18:39:21 -0000 On Sun, Feb 28, 2010 at 6:09 AM, Erik Scholtz, ArgonSoft GmbH wrote: > Hi, > hi > I did some research the last two weeks on how to build a cluster filesystem > on FreeBSD. > > My solution at the moment is, to rsync all filesystems once a minute, which > is rather to rare. So I tried to get a hook with KQueue to rsync the > filesystems on data-change. Unfortunatly I could not find a working solution > (had a try with IO::KQueue using perl). > i use rsync to make partial data backup .. ie: /etc, /usr/local/etc, /usr/home, /var/logs ... > How do you guys solve this problem (of a shared filesystem with rw-option)? > Any hints are welcome, since I'm getting very frustrated at the moment. > there is a project named 'hast'[1] for a clustered filesystem .. it's being developed by pawel .. the project has some completed milestones, so you can get it from fbsd src svn tree .. hast can do clustered filesystem right now but it's not complete, so there is no stable yet other way is gmirror[2] + ggated .. with that you can get a raid1 over net solution .. but i think it's not prepared to be used as master-master soluction > Greetings, > Erik > okey, same [1] http://www.freebsdfoundation.org/project%20announcements.shtml#Pawel [2] http://www.freebsddiary.org/gmirror.php From owner-freebsd-cluster@FreeBSD.ORG Mon Mar 1 19:12:10 2010 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 999C3106566B for ; Mon, 1 Mar 2010 19:12:10 +0000 (UTC) (envelope-from doug@polands.org) Received: from hrndva-omtalb.mail.rr.com (hrndva-omtalb.mail.rr.com [71.74.56.125]) by mx1.freebsd.org (Postfix) with ESMTP id 519888FC22 for ; Mon, 1 Mar 2010 19:12:10 +0000 (UTC) Received: from hrndva-omtalb.mail.rr.com ([10.128.143.52]) by hrndva-qmta02.mail.rr.com with ESMTP id <20100301185243110.IUID1979@hrndva-qmta02.mail.rr.com> for ; Mon, 1 Mar 2010 18:52:43 +0000 X-Authority-Analysis: v=1.0 c=1 a=tH3TCQxNhl/vWwTfLVWeHw==:17 a=dwBwwg8NJ5-TyUHOnOIA:9 a=t2MhQW9mOcMmOmXw9uxwMWKHh5cA:4 a=0VRiniRuMFoA:10 X-Cloudmark-Score: 0 X-Originating-IP: 75.87.219.217 Received: from [75.87.219.217] ([75.87.219.217:62599] helo=haran.polands.org) by hrndva-oedge02.mail.rr.com (envelope-from ) (ecelerity 2.2.2.39 r()) with ESMTP id 27/F1-20266-4BC0C8B4; Mon, 01 Mar 2010 18:51:33 +0000 Received: from email.polands.org (ararat.polands.org [172.16.1.20]) by haran.polands.org (8.14.3/8.14.3) with ESMTP id o21IpUuh033429; Mon, 1 Mar 2010 12:51:30 -0600 (CST) (envelope-from doug@polands.org) Received: from 209.103.214.35 (SquirrelMail authenticated user djp) by email.polands.org with HTTP; Mon, 1 Mar 2010 12:51:30 -0600 Message-ID: In-Reply-To: References: <4B8A4EFF.9050207@argonsoft.de> Date: Mon, 1 Mar 2010 12:51:30 -0600 From: "Doug Poland" To: "Leinier Cruz Salfran" User-Agent: SquirrelMail/1.4.20-RC2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: freebsd-cluster@freebsd.org Subject: Re: Cluster Filesystem on FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Mar 2010 19:12:10 -0000 On Mon, March 1, 2010 12:11, Leinier Cruz Salfran wrote: > On Sun, Feb 28, 2010 at 6:09 AM, Erik Scholtz, ArgonSoft GmbH > wrote: > >> I did some research the last two weeks on how to build a cluster >> filesystem on FreeBSD. >> >> My solution at the moment is, to rsync all filesystems once a >> minute, which is rather to rare. So I tried to get a hook with >> KQueue to rsync the filesystems on data-change. Unfortunatly I could >> not find a working solution (had a try with IO::KQueue using perl). >> > > i use rsync to make partial data backup .. ie: /etc, /usr/local/etc, > /usr/home, /var/logs ... > >> How do you guys solve this problem (of a shared filesystem with >> rw-option)? >> Any hints are welcome, since I'm getting very frustrated at the >> moment. >> > > there is a project named 'hast'[1] for a clustered filesystem .. it's > being developed by pawel .. the project has some completed milestones, > so you can get it from fbsd src svn tree .. hast can do clustered > filesystem right now but it's not complete, so there is no stable yet > > other way is gmirror[2] + ggated .. with that you can get a raid1 over > net solution .. but i think it's not prepared to be used as > master-master soluction > Neither hast nor gmirror+ggatd are cluster filesystems, in that only one "side" of the storage is available for writes at a point in time. Filesystems like OCFS2 and GFS allow multiple, simultaneous read-write access to block devices. Given there is not true cluster filesystem available for FreeBSD at this time, I wonder aloud why so many people are so quick to dismiss NFS? NFS provides "most" of features of a cluster filesystem today. If one were to choose NFS for shared storage, one could use tools available today to make NFS highly available (hast, gmirror+ggated). -- Regards, Doug From owner-freebsd-cluster@FreeBSD.ORG Tue Mar 2 09:48:28 2010 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7E1631065670 for ; Tue, 2 Mar 2010 09:48:28 +0000 (UTC) (envelope-from escholtz@argonsoft.de) Received: from coyote.quickmin.net (coyote.quickmin.net [217.14.112.24]) by mx1.freebsd.org (Postfix) with ESMTP id F12A58FC15 for ; Tue, 2 Mar 2010 09:48:27 +0000 (UTC) Received: (qmail 50935 invoked from network); 2 Mar 2010 10:48:26 +0100 Received: by simscan 1.4.0 ppid: 50932, pid: 50933, t: 0.0172s scanners: clamav: 0.95.2/m:49/d:8609 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=argonsoft.de; b=ckkr+hyTXygBS47yGwUbbRiDnJEX5u6ltpl4DnliAkCX4CrN+BJXtp3I+z0Yqil0pmxVUCL3RPt+lf6OYsTBaij0ZsXvcLoRsRL7ukvXwkHteGb7I0BuyS8kcBVN2wM+edzK8LLCllPzuZ3KNyq8k1jw8XosGU9qKvG7Qkl0g6I= ; Received: from dialbs-213-023-239-142.static.arcor-ip.net (HELO erik-scholtzs-macbook-pro-15.local) (00000150@213.23.239.142) by coyote.quickmin.net with SMTP; 2 Mar 2010 10:48:26 +0100 Message-ID: <4B8CDEE9.9020004@argonsoft.de> Date: Tue, 02 Mar 2010 10:48:25 +0100 From: "Erik Scholtz, ArgonSoft GmbH" User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812) MIME-Version: 1.0 To: Doug Poland References: <4B8A4EFF.9050207@argonsoft.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-cluster@freebsd.org Subject: Re: Cluster Filesystem on FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Mar 2010 09:48:28 -0000 Doug, have you ever tried to put a NFS-volume into the working-path of an apache-server? I did, and I could messure a significant performance-loss of the webserver by 1sec per page-load. The NFS was mounted on a 1GBit dedicated connection between the webserver and the nfs-server; I tried several TCP-options and MTU-Settings without any important change on the performance-loss. Compared to a dedicated mounted ISCSI-Volume this 1sec loss is a lot! I think NFS is great for changing big amounts of data. But for short read/write-access NFS does not seem to be the first choice. Greetings, Erik -- My blog: http://blog.elitecoderz.net Doug Poland wrote: > On Mon, March 1, 2010 12:11, Leinier Cruz Salfran wrote: >> On Sun, Feb 28, 2010 at 6:09 AM, Erik Scholtz, ArgonSoft GmbH >> wrote: >> >>> I did some research the last two weeks on how to build a cluster >>> filesystem on FreeBSD. >>> >>> My solution at the moment is, to rsync all filesystems once a >>> minute, which is rather to rare. So I tried to get a hook with >>> KQueue to rsync the filesystems on data-change. Unfortunatly I could >>> not find a working solution (had a try with IO::KQueue using perl). >>> >> i use rsync to make partial data backup .. ie: /etc, /usr/local/etc, >> /usr/home, /var/logs ... >> >>> How do you guys solve this problem (of a shared filesystem with >>> rw-option)? >>> Any hints are welcome, since I'm getting very frustrated at the >>> moment. >>> >> there is a project named 'hast'[1] for a clustered filesystem .. it's >> being developed by pawel .. the project has some completed milestones, >> so you can get it from fbsd src svn tree .. hast can do clustered >> filesystem right now but it's not complete, so there is no stable yet >> >> other way is gmirror[2] + ggated .. with that you can get a raid1 over >> net solution .. but i think it's not prepared to be used as >> master-master soluction >> > Neither hast nor gmirror+ggatd are cluster filesystems, in that only > one "side" of the storage is available for writes at a point in time. > Filesystems like OCFS2 and GFS allow multiple, simultaneous read-write > access to block devices. > > Given there is not true cluster filesystem available for FreeBSD at > this time, I wonder aloud why so many people are so quick to dismiss > NFS? NFS provides "most" of features of a cluster filesystem today. > If one were to choose NFS for shared storage, one could use tools > available today to make NFS highly available (hast, gmirror+ggated). > > >