Date: Sun, 28 Feb 2010 23:00:07 +0000 From: Luke Marsden <luke@digital-crocus.com> To: "Erik Scholtz, ArgonSoft GmbH" <escholtz@argonsoft.de> Cc: freebsd-cluster@freebsd.org Subject: Re: Cluster Filesystem on FreeBSD Message-ID: <1267398007.16514.89.camel@behemoth> In-Reply-To: <4B8A4EFF.9050207@argonsoft.de> References: <4B8A4EFF.9050207@argonsoft.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Erik, The best way I've found to monitor for filesystem activity on FreeBSD is with the audit_control kernel option. Recompile your kernel if necessary and pop this into /etc/security/audit_control (owned by root:wheel and 0600 perms): dir:/var/audit flags:fc,fd,fw minfree:20 naflags:lo policy:cnt filesz:0 Then this Python script (easily ported to Perl if you wish) will do the job of printing a path to stdout whenever a file is modified (and also pruning the praudit logfiles in /var/audit which aren't desirable for this application - we couldn't find a nicer way to do this): import popen2, sys, os pipe = popen2.Popen4("/usr/sbin/praudit /dev/auditpipe") count = 0 while True: line = pipe.fromchild.readline() if line=='': break # so that we die when praudit does if 'header' in line: header=line if '/path-to-monitor/' not in line: # useful to avoid a flood continue if 'path' in line: count += 1 if count % 1000 == 0: # clear out audit files to save disk space os.system('for X in /var/audit/*; do echo "" > "$X"; done') if 'fcntl' not in header: if line.split(',')[0].strip() == 'path': print "MODIFIED: "+line.split(',')[1].strip() sys.stdout.flush() Hope you find this useful, it's one small part of a FreeBSD-based web cluster product I'm working on, select parts of which we plan to release under the BSD license in the next few months in order to hopefully attract a developer community of FreeBSD cluster hackers. If you make any improvements to the script I'd appreciate getting them, just send them in to luke@hybrid-logic.co.uk Regarding filesystem replication (and without giving too much away) I recommend the ZFS filesystem with its fast, atomic snapshots and zfs send / zfs recv to do the job of rsync much more efficiently and at the block level. You won't find multi-master read/write with this solution on its own though. It is more useful to implement "streaming backups". At some point I'm planning to combine this approach with NFS failover in order to make every filesystem writable on every machine at all times, although this is not a requirement for our application right now. Stay tuned to www.hybrid-logic.co.uk for a timeline for open source availability of more of our cluster technology. (That part of the site isn't ready just yet, but should be in a few weeks.) Best Regards, Luke Marsden Hybrid Logic Ltd. On Sun, 2010-02-28 at 12:09 +0100, Erik Scholtz, ArgonSoft GmbH wrote: > Hi, > > I did some research the last two weeks on how to build a cluster > filesystem on FreeBSD. > > Since neither GFS ( http://sources.redhat.com/cluster/gfs/ ) nor OCFS2 ( > http://oss.oracle.com/projects/ocfs2/ ) are ported to FreeBSD, I took a > look at the GlusterFS-project ( http://www.gluster.org/ ), which also > does not compile on FreeBSD 7.x, as well as PVFS ( http://www.pvfs.org/ ). > > My solution at the moment is, to rsync all filesystems once a minute, > which is rather to rare. So I tried to get a hook with KQueue to rsync > the filesystems on data-change. Unfortunatly I could not find a working > solution (had a try with IO::KQueue using perl). > > NFS isn't really a solution, since it is slow (even on 1GBit) and is a > single point of failure: Clustering the NFS server with machines in > stand-by, that do a fail-over on a crash of the master-NFS is unstable > and makes a lot more costs to the project. > > I'm with FreeBSD since the 2.1 release and I really love this system for > it's performance and stability. But in the last year I came very often > to a point, where I had no solution on FreeBSD and several solutions for > Linux (like SAS-support, ...). > > > How do you guys solve this problem (of a shared filesystem with > rw-option)? Any hints are welcome, since I'm getting very frustrated at > the moment. > > Greetings, > Erik >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1267398007.16514.89.camel>