From owner-freebsd-cluster@FreeBSD.ORG Sun Feb 28 23:21:43 2010 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3A991065670 for ; Sun, 28 Feb 2010 23:21:43 +0000 (UTC) (envelope-from luke@digital-crocus.com) Received: from mail.digital-crocus.com (node2.digital-crocus.com [91.209.244.128]) by mx1.freebsd.org (Postfix) with ESMTP id 5C2BB8FC14 for ; Sun, 28 Feb 2010 23:21:43 +0000 (UTC) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=dkselector; d=digital-crocus.com; h=Received:Received:Subject:From:To:Cc:In-Reply-To:References:Content-Type:Date:Message-Id:Mime-Version:X-Mailer:Content-Transfer-Encoding:X-Spam-Score:X-Digital-Crocus-Maillimit:X-Authenticated-Sender:X-Complaints:X-Admin:X-Abuse; b=ogY1yE2PK36FkkJg0zjdH65zhztqH0Kp90D+oG5hGFHLEtElfFrsAa3j1Ldtlx4hJRjxlFfx5Y7SV+R5Uumy+FBRxxGKU4CIeXL2GOMZ6gwCs2GCzxOFsDHMtQiyW6Hy; Received: from luke by mail.digital-crocus.com with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Nls65-000E1V-5x for freebsd-cluster@freebsd.org; Sun, 28 Feb 2010 22:58:49 +0000 Received: from 127cr.net ([78.105.122.99] helo=[192.168.1.20]) by mail.digital-crocus.com with esmtpa (Exim 4.69 (FreeBSD)) (envelope-from ) id 1Nls64-000E1D-Ap; Sun, 28 Feb 2010 22:58:49 +0000 From: Luke Marsden To: "Erik Scholtz, ArgonSoft GmbH" In-Reply-To: <4B8A4EFF.9050207@argonsoft.de> References: <4B8A4EFF.9050207@argonsoft.de> Content-Type: text/plain Date: Sun, 28 Feb 2010 23:00:07 +0000 Message-Id: <1267398007.16514.89.camel@behemoth> Mime-Version: 1.0 X-Mailer: Evolution 2.26.1 Content-Transfer-Encoding: 7bit X-Spam-Score: -1.4 X-Digital-Crocus-Maillimit: done X-Authenticated-Sender: luke X-Complaints: abuse@digital-crocus.com X-Admin: admin@digital-crocus.com X-Abuse: abuse@digital-crocus.com (Please include full headers in abuse reports) Cc: freebsd-cluster@freebsd.org Subject: Re: Cluster Filesystem on FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Feb 2010 23:21:43 -0000 Hi Erik, The best way I've found to monitor for filesystem activity on FreeBSD is with the audit_control kernel option. Recompile your kernel if necessary and pop this into /etc/security/audit_control (owned by root:wheel and 0600 perms): dir:/var/audit flags:fc,fd,fw minfree:20 naflags:lo policy:cnt filesz:0 Then this Python script (easily ported to Perl if you wish) will do the job of printing a path to stdout whenever a file is modified (and also pruning the praudit logfiles in /var/audit which aren't desirable for this application - we couldn't find a nicer way to do this): import popen2, sys, os pipe = popen2.Popen4("/usr/sbin/praudit /dev/auditpipe") count = 0 while True: line = pipe.fromchild.readline() if line=='': break # so that we die when praudit does if 'header' in line: header=line if '/path-to-monitor/' not in line: # useful to avoid a flood continue if 'path' in line: count += 1 if count % 1000 == 0: # clear out audit files to save disk space os.system('for X in /var/audit/*; do echo "" > "$X"; done') if 'fcntl' not in header: if line.split(',')[0].strip() == 'path': print "MODIFIED: "+line.split(',')[1].strip() sys.stdout.flush() Hope you find this useful, it's one small part of a FreeBSD-based web cluster product I'm working on, select parts of which we plan to release under the BSD license in the next few months in order to hopefully attract a developer community of FreeBSD cluster hackers. If you make any improvements to the script I'd appreciate getting them, just send them in to luke@hybrid-logic.co.uk Regarding filesystem replication (and without giving too much away) I recommend the ZFS filesystem with its fast, atomic snapshots and zfs send / zfs recv to do the job of rsync much more efficiently and at the block level. You won't find multi-master read/write with this solution on its own though. It is more useful to implement "streaming backups". At some point I'm planning to combine this approach with NFS failover in order to make every filesystem writable on every machine at all times, although this is not a requirement for our application right now. Stay tuned to www.hybrid-logic.co.uk for a timeline for open source availability of more of our cluster technology. (That part of the site isn't ready just yet, but should be in a few weeks.) Best Regards, Luke Marsden Hybrid Logic Ltd. On Sun, 2010-02-28 at 12:09 +0100, Erik Scholtz, ArgonSoft GmbH wrote: > Hi, > > I did some research the last two weeks on how to build a cluster > filesystem on FreeBSD. > > Since neither GFS ( http://sources.redhat.com/cluster/gfs/ ) nor OCFS2 ( > http://oss.oracle.com/projects/ocfs2/ ) are ported to FreeBSD, I took a > look at the GlusterFS-project ( http://www.gluster.org/ ), which also > does not compile on FreeBSD 7.x, as well as PVFS ( http://www.pvfs.org/ ). > > My solution at the moment is, to rsync all filesystems once a minute, > which is rather to rare. So I tried to get a hook with KQueue to rsync > the filesystems on data-change. Unfortunatly I could not find a working > solution (had a try with IO::KQueue using perl). > > NFS isn't really a solution, since it is slow (even on 1GBit) and is a > single point of failure: Clustering the NFS server with machines in > stand-by, that do a fail-over on a crash of the master-NFS is unstable > and makes a lot more costs to the project. > > I'm with FreeBSD since the 2.1 release and I really love this system for > it's performance and stability. But in the last year I came very often > to a point, where I had no solution on FreeBSD and several solutions for > Linux (like SAS-support, ...). > > > How do you guys solve this problem (of a shared filesystem with > rw-option)? Any hints are welcome, since I'm getting very frustrated at > the moment. > > Greetings, > Erik >