Date: Mon, 21 Jun 2004 21:24:55 -0400 From: Bill Moran <wmoran@potentialtech.com> To: Chuck Swiger <cswiger@mac.com> Cc: questions@freebsd.org Subject: [OT] Re: What's the best possible email failover solution Message-ID: <20040621212455.44310df1.wmoran@potentialtech.com> In-Reply-To: <40D785A6.1040408@mac.com> References: <20040621132006.2b1a296f.wmoran@potentialtech.com> <a22ff294040621115173bad2e0@mail.gmail.com> <20040621172520.3544d6fe.wmoran@potentialtech.com> <20040621214348.GB63857@happy-idiot-talk.infracaninophile.co.uk> <20040621175626.3e762448.wmoran@potentialtech.com> <40D76DA3.9090809@mac.com> <20040621203245.1f0e7444.wmoran@potentialtech.com> <40D785A6.1040408@mac.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Chuck Swiger <cswiger@mac.com> wrote: > Bill Moran wrote: > > Chuck Swiger <cswiger@mac.com> wrote: > [ ... ] > >> The latter uses one-message-per-file, and ought to work *much* better both in > >> terms of performance and stability, and in terms of playing nice with the way > >> rsync wants to back things up. > > > > Doesn't really matter. Fact is, the mail directories are something on the order > > of 3G. No matter how efficiently I store them, rsync is not going to be able > > to back them up fast enough to hit the level of redundancy I'm shooting for. > > You may well be right, as you aren't really talking about performing backups, > you're talking about creating a fully redundant storage which is kept > up-to-date in realtime. > > > Although Maildirs might work a little better, since I wouldn't have to stop > > the IMAP server during backup. > > That, and the granularity of one-message-per-file fits perfectly with rsync's > file-driven model. I don't know about that. Last I checked, many files resulted in rsync taking a lot of time, and a lot of memory to build a file list. I can't say whether the end result is more efficient or not, as I've never tested it, but rsync does intelligently move portions of files, instead of the whole file, when changes occur. > > It takes about 30 minutes to rsync the system to the backup server right now. > > That's perfectly acceptable for nightly backup purposes. This is a 1.5Ghz > > with 256M RAM and 80G ATA 100 HDDs. If the system runs rysnc continuously > > 24/7, I still have 30mins old data. > > Oh, yes. Just don't forget that if you do eliminate this time gap, you still > ought to have another system actually taking backups. Any change the system > encounters will be replicated to the redundant mail storage system in real > time, including bad changes. Hehe ... I've been trying to explain that to some of my less intelligent clients for a while now ... "Yes, when you do something wrong, it backs that up as well ..." Actually had a client ask me once why the backup didn't know the difference between something it should be backing up, and something that shouldn't be backed up. I told him if I knew how to do that, I'd be a lot richer! > >>[ I don't think that stuffing email into a database is a particularly good > >>idea since that means keeping large blobs of non-relational data floating > >>around, something that the filesystem can do a better job of handling... ] > > > > It's a good idea if I want real-time redundancy. I see where you're coming > > from, and it's true that a RDBMS isn't the best way to store emails. But, > > when you look at the features available, it's the best way for this > > circumstance. With something like Slony, I'd have real-time redundancy > > with (I'm expecting) only a minor performance drop. Although I can't be > > sure until I can put something together to test. Reliability is much more > > important than performance in this case. Who cares if their email takes > > and extra 60 seconds to deliver, as long as it doesn't get lost! If the > > email arrives fast, it's useless if the server fails and the email is > > lost because the SMTP server told the delivering server that it had > > arrived and then crashed before it could be backed up. > > I suspect that the relatively heavy weight of database transactions compared > with filesystem access is going to slow things down a fair amount, too, > particularly when running against a replicated DB. But reliability over > performance is a fine choice to make. :-) > > Using RAID improves fault-tolerance, but you still end up with a > single-point-of-failure at the system level; using database replication gives > you higher availability, which seems to be what you mean when you talk about > "reliability". Perhaps SAN or NAS concepts might be worth considering, as you > can set up a fully-redundant fibre channel configuration where the storage is > shared between two or more systems, thus with no single-point-of-failure. Problem there is cost ... that hardware is a bit pricey, compared to PC hardware, anyway. -- Bill Moran Potential Technologies http://www.potentialtech.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040621212455.44310df1.wmoran>