Date: Mon, 26 Sep 2005 07:46:09 -0500 From: Eric Anderson <anderson@centtech.com> To: filip wuytack <filip@wuytack.net> Cc: freebsd-isp@freebsd.org, freebsd-cluster@freebsd.org Subject: Re: Options for synchronising filesystems Message-ID: <4337ED91.8080200@centtech.com> In-Reply-To: <4337E8A7.6070107@wuytack.net> References: <20050924141025.GA1236@uk.tiscali.com> <4337DF56.6030407@centtech.com> <4337E8A7.6070107@wuytack.net>
next in thread | previous in thread | raw e-mail | index | archive | help
filip wuytack wrote: > > > Eric Anderson wrote: > >> Brian Candler wrote: >> >>> Hello, >>> >>> I was wondering if anyone would care to share their experiences in >>> synchronising filesystems across a number of nodes in a cluster. I >>> can think >>> of a number of options, but before changing what I'm doing at the >>> moment I'd >>> like to see if anyone has good experiences with any of the others. >>> >>> The application: a clustered webserver. The users' CGIs run in a chroot >>> environment, and these clearly need to be identical (otherwise a CGI >>> running >>> on one box would behave differently when running on a different box). >>> Ultimately I'd like to synchronise the host OS on each server too. >>> >>> Note that this is a single-master, multiple-slave type of filesystem >>> synchronisation I'm interested in. >>> >>> >>> 1. Keep a master image on an admin box, and rsync it out to the >>> frontends >>> ------------------------------------------------------------------------- >>> >>> >>> This is what I'm doing at the moment. Install a master image in >>> /webroot/cgi, add packages there (chroot /webroot/cgi pkg_add ...), and >>> rsync it. [Actually I'm exporting it using NFS, and the frontends run >>> rsync >>> locally when required to update their local copies against the NFS >>> master] >>> >>> Disadvantages: >>> >>> - rsyncing a couple of gigs of data is not particularly fast, even >>> when only >>> a few files have changed >>> >>> - if a sysadmin (wrongly) changes a file on a front-end instead of on >>> the >>> master copy in the admin box, then the change will be lost when the next >>> rsync occurs. They might think they've fixed a problem, and then >>> (say) 24 >>> hours later their change is wiped. However if this is a config file, the >>> fact that the old file has been reinstated might not be noticed until >>> the >>> daemon is restarted or the box rebooted - maybe months later. This I >>> think >>> is the biggest fundamental problem. >>> >>> - files can be added locally and they will remain indefinitely >>> (unless we >>> use rsync --delete which is a bit scary). If this is done then adding >>> a new >>> machine into the cluster by rsyncing from the master will not pick up >>> these >>> extra files. >>> >>> So, here are the alternatives I'm considering, and I'd welcome any >>> additional suggestions too. >> >> >> >> Here's a few ideas on this: do multiple rsyncs, one for each top level >> directory. That might speed up your total rsync process. Another >> similar method is using a content revisioning system. This is only >> good for some cases, but something like subversion might work ok here. >> >> >> >>> 2. Run the images directly off NFS >>> ---------------------------------- >>> >>> I've had this running before, even the entire O/S, and it works just >>> fine. >>> However the NFS server itself then becomes a critical >>> single-point-of-failure: if it has to be rebooted and is out of >>> service for >>> 2 minutes, then the whole cluster is out of service for that time. >>> >>> I think this is only feasible if I can build a highly-available NFS >>> server, >>> which really means a pair of boxes serving the same data. Since the >>> system >>> image is read-only from the point of view of the frontends, this >>> should be >>> easy enough: >>> >>> frontends frontends >>> | | | | | | >>> NFS -----------> NFS >>> server 1 sync server 2 >>> >>> As far as I know, NFS clients don't support the idea of failing over >>> from >>> one server to another, so I'd have to make a server pair which >>> transparently >>> fails over. >>> >>> I could make one NFS server take over the other server's IP address >>> using >>> carp or vrrp. However, I suspect that the clients might notice. I >>> know that >>> NFS is 'stateless' in the sense that a server can be rebooted, but for a >>> client to be redirected from one server to the other, I expect that >>> these >>> filesytems would have to be *identical*, down to the level of the inode >>> numbers being the same. >>> >>> If that's true, then rsync between the two NFS servers won't cut it. >>> I was >>> thinking of perhaps using geom_mirror plus ggated/ggatec to make a >>> block-identical read-only mirror image on NFS server 2 - this also >>> has the >>> advantage that any updates are close to instantaneous. >>> >>> What worries me here is how NFS server 2, which has the mirrored >>> filesystem >>> mounted read-only, will take to having the data changed under its >>> nose. Does >>> it for example keep caches of inodes in memory, and what would happen if >>> those inodes on disk were to change? I guess I can always just >>> unmount and >>> remount the filesystem on NFS server 2 after each change. >> >> >> >> I've tried doing something similar. I used fiber attached storage, >> and had multiple hosts mounting the same partition. It seemed as >> though when host A mounted the filesystem read-write, and then host B >> mounted it read-only, any changes made by host A were not seen by B, >> and even remounting did not always bring it up to current state. I >> believe it has to do with the buffer cache and host A's desire to keep >> things (like inode changes, block maps, etc) in cache and not write >> them to disk. FreeBSD does not currently have a multi-system cache >> coherency protocol to distribute that information to other hosts. >> This is something I think would be very useful for many people. I >> suppose you could just mount the filesystem when you know a change has >> happened, but you still may not see the change. Maybe mounting the >> filesystem on host A with the sync option would help. >> >>> My other concern is about susceptibility to DoS-type attacks: if one >>> frontend were to go haywire and start hammering the NFS servers >>> really hard, >>> it could impact on all the other machines in the cluster. >>> >>> However, the problems of data synchronisation are solved: any change >>> made on >>> the NFS server is visible identically to all front-ends, and >>> sysadmins can't >>> make changes on the front-ends because the NFS export is read-only. >> >> >> >> This was my first thought too, and a highly available NFS server is >> something any NFS heavy installation wants (needs). There are a few >> implementations of clustered filesystems out there, but non for >> FreeBSD (yet). What that allows is multiple machines talking to a >> shared storage with read/write access. Very handy, but since you only >> need read-only access, I think your problem is much simpler, and you >> can get away with a lot less. >> >> >>> 3. Use a network distributed filesystem - CODA? AFS? >>> ---------------------------------------------------- >>> >>> If each frontend were to access the filesystem as a read-only network >>> mount, >>> but have a local copy to work with in the case of disconnected >>> operation, >>> then the SPOF of an NFS server would be eliminated. >>> >>> However, I have no experience with CODA, and although it's been in >>> the tree >>> since 2002, the README's don't inspire confidence: >>> >>> "It is mostly working, but hasn't been run long enough to be sure >>> all the >>> bugs are sorted out. ... This code is not SMP ready" >>> >>> Also, a local cache is no good if the data you want during disconnected >>> operation is not in the cache at that time, which I think means this >>> idea is >>> not actually a very good one. >> >> >> >> There is also a port for coda. I've been reading about this, and >> it's an interesting filesystem, but I'm just not sure of it's >> usefulness yet. >> >> >>> 4. Mount filesystems read-only >>> ------------------------------ >>> >>> On each front-end I could store /webroot/cgi on a filesystem mounted >>> read-only to prevent tampering (as long as the sysadmin doesn't >>> remount it >>> read-write of course). That would work reasonably well, except that >>> being >>> mounted read-only I couldn't use rsync to update it! >>> >>> It might also work with geom_mirror and ggated/ggatec, except for the >>> issue >>> I raised before about changing blocks on a filesystem under the nose >>> of a >>> client who is actively reading from it. >> >> >> >> I suppose you could mount r/w only when doing the rsync, then switch >> back to ro once complete. You should be able to do this online, >> without any issues or taking the filesystem offline. >> >> >>> 5. Using a filesystem which really is read-only >>> ----------------------------------------------- >>> >>> Better tamper-protection could be had by keeping data in a filesystem >>> structure which doesn't support any updates at all - such as cd9660 or >>> geom_uzip. >>> >>> The issue here is how to roll out a new version of the data. I could >>> push >>> out a new filesystem image into a second partition, but it would then be >>> necessary to unmount the old filesystem and remount the new on the same >>> place, and you can't really unmount a filesystem which is in use. So >>> this >>> would require a reboot. >>> >>> I was thinking that some symlink trickery might help: >>> >>> /webroot/cgi -> /webroot/cgi1 >>> /webroot/cgi1 # filesystem A mounted here >>> /webroot/cgi2 # filesystem B mounted here >>> >>> It should be possible to unmount /webroot/cgi2, dd in a new image, >>> remount >>> it, and change the symlink to point to /webroot/cgi2. After a little >>> while, >>> hopefully all the applications will stop using files in >>> /webroot/cgi1, so >>> this one can be unmounted and a new one put in its place on the next >>> update. >>> However this is not guaranteed, especially if there are long-lived >>> processes >>> using binary images in this partition. You'd still have to stop and >>> restart >>> all those processes. >>> >>> If reboots were acceptable, then the filesystem image could also be >>> stored >>> in ramdisk pulled in via pxeboot. This makes sense especially for >>> geom_uzip >>> where the data is pre-compressed. However I would still prefer to avoid >>> frequent reboots if at all possible. Also, whilst a ramdisk might be >>> OK for >>> the root filesystem, a typical CGI environment (with perl, php, ruby, >>> python, and loads of libraries) would probably be too large anyway. >>> >>> >>> 6. Journaling filesystem replication >>> ------------------------------------ >>> >>> If the data were stored on a journaling filesystem on the master box, >>> and >>> the journal logs were distributed out to the slaves, then they would all >>> have identical filesystem copies and only a minimal amount of data would >>> need to be pushed out to each machine on each change. (This would be >>> rather >>> like NetApps and their snap-mirroring system). However I'm not aware >>> of any >>> journaling filesystem for FreeBSD, let alone whether it would support >>> filesystem replication in this way. >> >> >> >> There is a project underway for UFSJ (UFS journaling). Maybe once it >> is complete, and bugs are ironed out, one could implement a journal >> distribution piece to send the journal updates to multiple hosts and >> achieve what you are thinking, however, that only distributes the >> meta-data, and not the actual data. >> >> > Have a look at dragonfly BSD for this. They are working on a journaling > filesystem that will do just that. Do you have a link to some information on this? I've been looking at Dragonfly, but I'm having trouble finding good information on what is already working, in planning, etc. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4337ED91.8080200>