Date: Fri, 13 Mar 1998 12:36:40 +0000 From: Alan Judge <Alan.Judge@indigo.ie> To: freebsd-isp@FreeBSD.ORG Subject: Re: Scalable mail server Message-ID: <199803131236.EAA17875@hub.freebsd.org> In-Reply-To: Message from "Andras Tudos - Computronic, C3" dated Thursday at 21:31.
next in thread | raw e-mail | index | archive | help
Andras> we have to scale our qmail based mail server (with a custom Andras> made www frontend) to multiple machines to be able to serve Andras> ~100000 users and more. Currently we have a single server Andras> based heavily overloaded system with 30000+ users. We have Andras> many plans, but I would prefer to hear how others would start Andras> to construct such a system. We build all the machines ourselfs Andras> (including PII based PCs, RAID, network) and write the Andras> necessary code. Andras> All ideas, especially real world experiences are warmly Andras> welcome... (Especially file-system, user database and Andras> redundancy issues are critical. Would you use SMP with dual Andras> PIIs or 2.2.5? NFS issues? Postgres instead of passwd? Andras> Round-robin DNS or something else?) We're doing exactly this in production. Four PPro boxes running 2.2-STABLE and qmail (as well as Apache for user web access); all the data is stored on a NetApp filer, with nothing but OS stuff and the qmail mail queues on local disk. We use Maildir format for storing messages, so NFS locking is not an issue. Works perfectly and has very good redundancy. No downtime at all since we went live last November. We can add a new front-end machine in about an hour or so. With more front-ends and bigger filers available, I don't see any problems scaling to at least a few hundred thousand mailboxes or more. We have maybe 30K now and the load on the machines and the filer is very low. Each machine has two network interfaces. One to the public and one onto a private network for access to the filer using NFS. Everything hooked up with a Catalyst 5000. I haven't tried out the SMP stuff yet, but even PPro boxes are plenty fast for this sort of stuff. We use round-robin DNS across three of the machines. The fourth is a hot spare that can be made live at any stage. We keep all the machines in sync using ssh and rdist. We still use passwd databases at the moment, though even with increases in the DB cache size, pwd_mkdb takes up to a minute. The only trick is if you want anti-spamming support to work, yet still want users to be able to roam onto other networks. To get around this, we made mods to qmail so that when users authenticate via POP, the details are multicast to all the other servers to be used to dynamically change the list of IP addresses allowed to relay SMTP. -- Alan Judge Phone: +353-1-6046901 Indigo Internet Services Fax: +353-1-6046948 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-isp" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199803131236.EAA17875>