From owner-freebsd-isp Wed Jul 25 4:44: 3 2001 Delivered-To: freebsd-isp@freebsd.org Received: from jake.akitanet.co.uk (jake.akitanet.co.uk [212.1.130.131]) by hub.freebsd.org (Postfix) with ESMTP id 2864F37B401 for ; Wed, 25 Jul 2001 04:43:51 -0700 (PDT) (envelope-from wiggy@wopr.akitanet.co.uk) Received: from dsl-212-135-208-201.dsl.easynet.co.uk ([212.135.208.201] helo=wopr.akitanet.co.uk) by jake.akitanet.co.uk with esmtp (Exim 3.13 #3) id 15PN3E-000H8L-00; Wed, 25 Jul 2001 12:42:04 +0100 Received: from wiggy by wopr.akitanet.co.uk with local (Exim 3.21 #2) id 15PN50-00034t-00; Wed, 25 Jul 2001 12:43:54 +0100 Date: Wed, 25 Jul 2001 12:43:54 +0100 From: Paul Robinson To: Gabriel Ambuehl Cc: Enriko Groen , 'Tony Saign' , freebsd-isp@freebsd.org Subject: Re: Redundant setup on a budget?? Message-ID: <20010725124353.A6548@jake.akitanet.co.uk> References: <510EAC2065C0D311929200A0247252622F7A7B@NETIVITY-FS> <20010724154211.C34017@jake.akitanet.co.uk> <1241681557.20010725114735@buz.ch> <20010725112250.N83511@jake.akitanet.co.uk> <1996903256.20010725131437@buz.ch> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1996903256.20010725131437@buz.ch>; from gabriel_ambuehl@buz.ch on Wed, Jul 25, 2001 at 01:14:37PM +0200 X-Scanner: exiscan *15PN3E-000H8L-00*$AK$ur3a.Om7nWOFKFtcPl//e.* Sender: owner-freebsd-isp@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On Jul 25, Gabriel Ambuehl wrote: > It's well hidden in there. Generally, I'd prefer ipfilter over ipfw > any time, IMHO it's the much more sophisticated package but it > sometimes suffers a bit of it's extrem multi platform approach, which > means that you can't always use the newest release on your box. I prefer ipfw simply because I have more experience with it. ipfilter is a little bit too Linuxy in it's approach. > IMNSHO, this is a flawed approach as this one filesystem which must > reside on one box if you don't use distributed RAID, is THE single > point of failure. And with distributed RAID you haven't addressed your problem of atomic transactions. It's all very well having no single point of failure, but I can guarantee in every scenario you will have multiple single points of failure. "You're using the same OS on every machine?", "You use the same power company and brand of UPS and generator for all your power?", etc. It can all come back to single points of failure. In my experience, I would rather have multiple RAID cards, in a beefy box, perhaps doing incremental backups once an hour to a hot standby, with hot-swap disks, etc. and know that my file writes are being locked properly than take a risk with several boxes informing each other of transactions and just hoping it works. The problem with your approach, is that it is likely to look as though it's working fine at first, but once you put load on, maybe 0.00001% of transactions will start suffering. Then 0.000015%, and so on as the load increases. It will just look like a weird bug somewhere down in the system that will be easy to pass over. The occasional screw-up. But as load increases, these problems will rise steadily. > Some problems with this: rpc.lockd isn't one of the fastest pieces of > code I can think of (can't be, it's whole purpose doesn't allow it). > Further, if you have one filesystem, then you also got the problem > that if one machine is cracked, the cracker can fuck with all of > your data. On a decent mirroring scheme, this shouldn't be the case > (rather easy if you got different sets of data you can put into their > own segments, a bit harder if the data must be available to each and > every node of your cluster but this isn't the case for us, as we do > webhosting which can very nicely be segmented). Well, I wasn't going to allow all machines access to all the data. If he gets into an SQL server, he gets to mess with SQL data. He gets into a web server, he gets access to web data. However, I've already had discussions about security in general on this and other lists, and I don't want to re-visit them now. Although rpc.lockd isn't fast, you have to ask the question as to whether speed is what is imporant to you in this environment. In a web-hosting environment, we're talking about a heavy-read setup. We're not going to be too worried. For SQL stuff, we might get concerned if we're doing a lot of INSERTs and UPDATEs. For a mail setup, we are definitely going to be concerned. However, can you take the risk with your customer's mail that because you haven't got locking sorted that mail is being written to a spool from one machine, but then gets trashed by mail from another machine? no? In that case you need locking. > Trunking isn't supported at all by FreeBSD if I'm not totally > mistaken. That's why I put a smiley after my statement. Trunking is hard. It'd be nice to have, but it's hard. So, off to see what Gigabit cards FBSD is supporting now. :-) > If you got atomic operations on the filesystem, you're doing > something > wrong, IMO. That's what databases are for. And if you have several MySQL servers acting as heads to the same database? You need file-level locking if your cluster is to have any write operations. To say file servers shouldn't have atomic locking raises the question as to why the hell qpopper puts locks in place. To me, it's obvious, that servers are EXACTLY where atomic actions should be taking place. > If you need real load balancing, use a DBMS, that's what were made > for. I didn't say I needed it. I just said I was going to build it. You can't see the advantage of being able to cluster free SQL servers together? You can't see how docs on how to get multiple SMTP/POP3/IMAP servers all working on the same spools on a big fat RAID is not useful? Fine, you don't want it. Others do. I'm planning on doing it because I work in a job where I get to do fun things, the way I want to. :-) > More or less ACK. Mostly FTP uploads by the users and some writes to > the FS from some badly implemented scripts which I'm not going to > babysit. If you Whereas we do full-on scripting/e-com/god knows what where we are doing read-write all the time. For your setup, what I'm suggesting is overkill. > Why do you need NFS locking, then? MySQL does everything that is > needed to replicate the data. It gets hairy, though, if one DB server > isn't enough to cope with the update/insert statements but then you > should probably spend more money on the DB server. I'd rather have 10 boxes costing my 500 quid each than 1 box costing me 20k. So would a lot of other companies. Plus, it's more scalable. Plus, I'm doing this because it's fun. :-) > First point to get clear about it is: does your project allow it to > be > run entirely out of a DB (perhaps with local, non critical FS caches > which don't matter if the go down), if it can (and most DB based apps > can, problems starts with flatfile based scripts), you're lucky, as > you simply need to have some NAT gateway (two would be better, of > course, so you haven't got a single point of failure there) and some > backend DB servers which do their replication themselves anyway. The problem with replication, is we get into trouble with atomic actions again. Anyway, it's still early days, and I'm still thinking of different approaches. -- Paul Robinson ,--------------------------------------- Technical Director @ Akita | A computer lets you make more mistakes PO Box 604, Manchester, M60 3PR | than any other invention with the T: +44 (0) 161 228 6388 (F:6389)| possible exceptions of handguns and | Tequila - Mitch Ratcliffe `----- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message