Date: Sun, 28 Feb 2010 23:00:07 +0000 From: Luke Marsden <luke@digital-crocus.com> To: "Erik Scholtz, ArgonSoft GmbH" <escholtz@argonsoft.de> Cc: freebsd-cluster@freebsd.org Subject: Re: Cluster Filesystem on FreeBSD Message-ID: <1267398007.16514.89.camel@behemoth> In-Reply-To: <4B8A4EFF.9050207@argonsoft.de> References: <4B8A4EFF.9050207@argonsoft.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Erik,
The best way I've found to monitor for filesystem activity on FreeBSD is
with the audit_control kernel option. Recompile your kernel if necessary
and pop this into /etc/security/audit_control (owned by root:wheel and
0600 perms):
dir:/var/audit
flags:fc,fd,fw
minfree:20
naflags:lo
policy:cnt
filesz:0
Then this Python script (easily ported to Perl if you wish) will do the
job of printing a path to stdout whenever a file is modified (and also
pruning the praudit logfiles in /var/audit which aren't desirable for
this application - we couldn't find a nicer way to do this):
import popen2, sys, os
pipe = popen2.Popen4("/usr/sbin/praudit /dev/auditpipe")
count = 0
while True:
line = pipe.fromchild.readline()
if line=='':
break # so that we die when praudit does
if 'header' in line:
header=line
if '/path-to-monitor/' not in line: # useful to avoid a flood
continue
if 'path' in line:
count += 1
if count % 1000 == 0: # clear out audit files to save disk space
os.system('for X in /var/audit/*; do echo "" > "$X"; done')
if 'fcntl' not in header:
if line.split(',')[0].strip() == 'path':
print "MODIFIED: "+line.split(',')[1].strip()
sys.stdout.flush()
Hope you find this useful, it's one small part of a FreeBSD-based web
cluster product I'm working on, select parts of which we plan to release
under the BSD license in the next few months in order to hopefully
attract a developer community of FreeBSD cluster hackers.
If you make any improvements to the script I'd appreciate getting them,
just send them in to luke@hybrid-logic.co.uk
Regarding filesystem replication (and without giving too much away) I
recommend the ZFS filesystem with its fast, atomic snapshots and zfs
send / zfs recv to do the job of rsync much more efficiently and at the
block level.
You won't find multi-master read/write with this solution on its own
though. It is more useful to implement "streaming backups". At some
point I'm planning to combine this approach with NFS failover in order
to make every filesystem writable on every machine at all times,
although this is not a requirement for our application right now.
Stay tuned to www.hybrid-logic.co.uk for a timeline for open source
availability of more of our cluster technology. (That part of the site
isn't ready just yet, but should be in a few weeks.)
Best Regards,
Luke Marsden
Hybrid Logic Ltd.
On Sun, 2010-02-28 at 12:09 +0100, Erik Scholtz, ArgonSoft GmbH wrote:
> Hi,
>
> I did some research the last two weeks on how to build a cluster
> filesystem on FreeBSD.
>
> Since neither GFS ( http://sources.redhat.com/cluster/gfs/ ) nor OCFS2 (
> http://oss.oracle.com/projects/ocfs2/ ) are ported to FreeBSD, I took a
> look at the GlusterFS-project ( http://www.gluster.org/ ), which also
> does not compile on FreeBSD 7.x, as well as PVFS ( http://www.pvfs.org/ ).
>
> My solution at the moment is, to rsync all filesystems once a minute,
> which is rather to rare. So I tried to get a hook with KQueue to rsync
> the filesystems on data-change. Unfortunatly I could not find a working
> solution (had a try with IO::KQueue using perl).
>
> NFS isn't really a solution, since it is slow (even on 1GBit) and is a
> single point of failure: Clustering the NFS server with machines in
> stand-by, that do a fail-over on a crash of the master-NFS is unstable
> and makes a lot more costs to the project.
>
> I'm with FreeBSD since the 2.1 release and I really love this system for
> it's performance and stability. But in the last year I came very often
> to a point, where I had no solution on FreeBSD and several solutions for
> Linux (like SAS-support, ...).
>
>
> How do you guys solve this problem (of a shared filesystem with
> rw-option)? Any hints are welcome, since I'm getting very frustrated at
> the moment.
>
> Greetings,
> Erik
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1267398007.16514.89.camel>