Date: Wed, 25 Jul 2001 12:43:54 +0100 From: Paul Robinson <paul@akita.co.uk> To: Gabriel Ambuehl <gabriel_ambuehl@buz.ch> Cc: Enriko Groen <enriko.groen@netivity.nl>, 'Tony Saign' <tony@saignon.net>, freebsd-isp@freebsd.org Subject: Re: Redundant setup on a budget?? Message-ID: <20010725124353.A6548@jake.akitanet.co.uk> In-Reply-To: <1996903256.20010725131437@buz.ch>; from gabriel_ambuehl@buz.ch on Wed, Jul 25, 2001 at 01:14:37PM %2B0200 References: <510EAC2065C0D311929200A0247252622F7A7B@NETIVITY-FS> <20010724154211.C34017@jake.akitanet.co.uk> <1241681557.20010725114735@buz.ch> <20010725112250.N83511@jake.akitanet.co.uk> <1996903256.20010725131437@buz.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jul 25, Gabriel Ambuehl <gabriel_ambuehl@buz.ch> wrote:
> It's well hidden in there. Generally, I'd prefer ipfilter over ipfw
> any time, IMHO it's the much more sophisticated package but it
> sometimes suffers a bit of it's extrem multi platform approach, which
> means that you can't always use the newest release on your box.
I prefer ipfw simply because I have more experience with it. ipfilter is
a little bit too Linuxy in it's approach.
> IMNSHO, this is a flawed approach as this one filesystem which must
> reside on one box if you don't use distributed RAID, is THE single
> point of failure.
And with distributed RAID you haven't addressed your problem of atomic
transactions. It's all very well having no single point of failure, but I
can guarantee in every scenario you will have multiple single points of
failure. "You're using the same OS on every machine?", "You use the same
power company and brand of UPS and generator for all your power?", etc. It
can all come back to single points of failure.
In my experience, I would rather have multiple RAID cards, in a beefy box,
perhaps doing incremental backups once an hour to a hot standby, with
hot-swap disks, etc. and know that my file writes are being locked properly
than take a risk with several boxes informing each other of transactions and
just hoping it works. The problem with your approach, is that it is likely
to look as though it's working fine at first, but once you put load on,
maybe 0.00001% of transactions will start suffering. Then 0.000015%, and so
on as the load increases. It will just look like a weird bug somewhere down
in the system that will be easy to pass over. The occasional screw-up. But
as load increases, these problems will rise steadily.
> Some problems with this: rpc.lockd isn't one of the fastest pieces of
> code I can think of (can't be, it's whole purpose doesn't allow it).
> Further, if you have one filesystem, then you also got the problem
> that if one machine is cracked, the cracker can fuck with all of
> your data. On a decent mirroring scheme, this shouldn't be the case
> (rather easy if you got different sets of data you can put into their
> own segments, a bit harder if the data must be available to each and
> every node of your cluster but this isn't the case for us, as we do
> webhosting which can very nicely be segmented).
Well, I wasn't going to allow all machines access to all the data. If he
gets into an SQL server, he gets to mess with SQL data. He gets into a web
server, he gets access to web data. However, I've already had discussions
about security in general on this and other lists, and I don't want to
re-visit them now.
Although rpc.lockd isn't fast, you have to ask the question as to whether
speed is what is imporant to you in this environment. In a web-hosting
environment, we're talking about a heavy-read setup. We're not going to be
too worried. For SQL stuff, we might get concerned if we're doing a lot of
INSERTs and UPDATEs. For a mail setup, we are definitely going to be
concerned. However, can you take the risk with your customer's mail that
because you haven't got locking sorted that mail is being written to a spool
from one machine, but then gets trashed by mail from another machine? no? In
that case you need locking.
> Trunking isn't supported at all by FreeBSD if I'm not totally
> mistaken.
That's why I put a smiley after my statement. Trunking is hard. It'd be nice
to have, but it's hard. So, off to see what Gigabit cards FBSD is supporting
now. :-)
> If you got atomic operations on the filesystem, you're doing
> something
> wrong, IMO. That's what databases are for.
And if you have several MySQL servers acting as heads to the same
database? You need file-level locking if your cluster is to have any write
operations. To say file servers shouldn't have atomic locking raises the
question as to why the hell qpopper puts locks in place. To me, it's
obvious, that servers are EXACTLY where atomic actions should be taking
place.
> If you need real load balancing, use a DBMS, that's what were made
> for.
I didn't say I needed it. I just said I was going to build it. You can't see
the advantage of being able to cluster free SQL servers together? You can't
see how docs on how to get multiple SMTP/POP3/IMAP servers all working on
the same spools on a big fat RAID is not useful? Fine, you don't want
it. Others do. I'm planning on doing it because I work in a job where I get
to do fun things, the way I want to. :-)
> More or less ACK. Mostly FTP uploads by the users and some writes to
> the FS from some badly implemented scripts which I'm not going to
> babysit. If you
Whereas we do full-on scripting/e-com/god knows what where we are doing
read-write all the time. For your setup, what I'm suggesting is overkill.
> Why do you need NFS locking, then? MySQL does everything that is
> needed to replicate the data. It gets hairy, though, if one DB server
> isn't enough to cope with the update/insert statements but then you
> should probably spend more money on the DB server.
I'd rather have 10 boxes costing my 500 quid each than 1 box costing me
20k. So would a lot of other companies. Plus, it's more scalable. Plus, I'm
doing this because it's fun. :-)
> First point to get clear about it is: does your project allow it to
> be
> run entirely out of a DB (perhaps with local, non critical FS caches
> which don't matter if the go down), if it can (and most DB based apps
> can, problems starts with flatfile based scripts), you're lucky, as
> you simply need to have some NAT gateway (two would be better, of
> course, so you haven't got a single point of failure there) and some
> backend DB servers which do their replication themselves anyway.
The problem with replication, is we get into trouble with atomic actions
again. Anyway, it's still early days, and I'm still thinking of different
approaches.
--
Paul Robinson ,---------------------------------------
Technical Director @ Akita | A computer lets you make more mistakes
PO Box 604, Manchester, M60 3PR | than any other invention with the
T: +44 (0) 161 228 6388 (F:6389)| possible exceptions of handguns and
| Tequila - Mitch Ratcliffe
`-----
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010725124353.A6548>