Date: Tue, 15 Jul 2008 11:47:57 -0400 From: Sven Willenberger <sven@dmv.com> To: Jeremy Chadwick <koitsu@FreeBSD.org> Cc: freebsd-stable@FreeBSD.org Subject: Re: Multi-machine mirroring choices Message-ID: <1216136877.27608.36.camel@lanshark.dmv.com> In-Reply-To: <20080715145426.GA31340@eos.sc1.parodius.com> References: <1216130834.27608.27.camel@lanshark.dmv.com> <20080715145426.GA31340@eos.sc1.parodius.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-yNnrZGq+GBL27dLqLdks Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Tue, 2008-07-15 at 07:54 -0700, Jeremy Chadwick wrote: > On Tue, Jul 15, 2008 at 10:07:14AM -0400, Sven Willenberger wrote: > > 3) The send/recv feature of zfs was something I had not even considered > > until very recently. My understanding is that this would work by a) > > taking a snapshot of master_data1 b) zfs sending that snapshot to > > slave_data1 c) via ssh on pipe, receiving that snapshot on slave_data1 > > and then d) doing incremental snapshots, sending, receiving as in > > (a)(b)(c). How time/cpu intensive is the snapshot generation and just > > how granular could this be done? I would imagine for systems with litle > > traffic/changes this could be practical but what about systems that may > > see a lot of files added, modified, deleted to the filesystem(s)? >=20 > I can speak a bit on ZFS snapshots, because I've used them in the past > with good results. >=20 > Compared to UFS2 snapshots (e.g. dump -L or mksnap_ffs), ZFS snapshots > are fantastic. The two main positives for me were: >=20 > 1) ZFS snapshots take significantly less time to create; I'm talking > seconds or minutes vs. 30-45 minutes. I also remember receiving mail > from someone (on -hackers? I can't remember -- let me know and I can > dig through my mail archives for the specific mail/details) stating > something along the lines of "over time, yes, UFS2 snapshots take > longer and longer, it's a known design problem". >=20 > 2) ZFS snapshots, when created, do not cause the system to more or less > deadlock until the snapshot is generated; you can continue to use the > system during the time the snapshot is being generated. While with > UFS2, dump -L and mksnap_ffs will surely disappoint you. >=20 > We moved all of our production systems off of using dump/restore solely > because of these aspects. We didn't move to ZFS though; we went with > rsync, which is great, except for the fact that it modifies file atimes > (hope you use Maildir and not classic mbox/mail spools...). >=20 > ZFS's send/recv capability (over a network) is something I didn't have > time to experiment with, but it looked *very* promising. The method is > documented in the manpage as "Example 12", and is very simple -- as it > should be. You don't have to use SSH either, by the way[1]. The examples do list ssh as the way of initiating the receiving end; I am curious as to what the alterative would be (short of installing openssh-portable and using cipher=3Dno). > One of the "annoyances" to ZFS snapshots, however, was that I had to > write my own script to do snapshot rotations (think incremental dump(8) > but using ZFS snapshots). That is what I was afraid of. Using snapshots would seem to involve a bit of housekeeping. Furthermore, it sounds more suited to a system that needs periodic rather than constant backing up (syncing). > > I would be interested to hear anyone's experience with any (or all) of > > these methods and caveats of each. I am leaning towards ggate(dc) + > > zpool at the moment assuming that zfs can "smartly" rebuild the mirror > > after the slave's ggated processes bug out. >=20 > I don't have any experience with GEOM gate, so I can't comment on it. > But I would highly recommend you discuss the shortcomings with pjd@, > because he definitely listens. >=20 > However, I must ask you this: why are you doing things the way you are? > Why are you using the equivalent of RAID 1 but for entire computers? Is > there some reason you aren't using a filer (e.g. NetApp) for your data, > thus keeping it centralised? There has been recent discussion of using > FreeBSD with ZFS as such, over on freebsd-fs. If you want a link to the > thread, I can point you to it. Basically I am trying to eliminate the "single point of failure". The project prior to this had such a failure that even a raid5 setup could not get out of. It was determined at that point that a single-machine storage solution would no longer suffice. What I am trying to achieve is having a slave machine that could take over as the file server in the event the master machine goes down. This could be something as simple as the master's network connection going down (CARP to the rescue on the slave) to a complete failure of the master. While zfs send/recv sounds like a good option for periodic backups, I don't think it will fit my purpose. Zpool or gmirror will be a better fit. I see in posts following my initial post that there is reference to improvements in ggate[cd] and/or tcp since 6.2 (and I have moved to 7.0 now) so that bodes well. The question then becomes a matter of which system would be easier to manage in terms of a) the master rebuilding the mirror in the event the slave goes down or ggate[cd] disconnects and b) have the slave become the master for serving files and mounting the drives that were part of the mirror. Thanks to the other posters, I see others are doing what I am trying to accomplish and there were some additional "tuneables" for ggate[cd] that I had not yet incorporated that were mentioned. Sven --=-yNnrZGq+GBL27dLqLdks Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) iD8DBQBIfMatSnmnd8q3JGsRApCCAKCJGbY0i+/b4mh9JePzvxemrafewgCdEJwO y38BCizs6I2MXUPPlcbBSf8= =1nDM -----END PGP SIGNATURE----- --=-yNnrZGq+GBL27dLqLdks--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1216136877.27608.36.camel>