Date: Thu, 5 May 2005 11:10:18 -0700 From: Benson Wong <tummytech@gmail.com> To: Odhiambo Washington <wash@wananchi.com>, FreeBSD <freebsd-questions@freebsd.org> Subject: Re: installing big qmail server ... where to start? Message-ID: <860807bf05050511103bd6412d@mail.gmail.com> In-Reply-To: <20050505083227.GA91192@ns2.wananchi.com> References: <200505041258.39573.haimat@lame.at> <860807bf0505041646120d4f18@mail.gmail.com> <20050505083227.GA91192@ns2.wananchi.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I'm CC this answer back to FBSD-Questions.=20 On 5/5/05, Odhiambo Washington <wash@wananchi.com> wrote: > * Benson Wong <tummytech@gmail.com> [20050505 02:56]: wrote: >=20 > Hi Ben, >=20 >=20 > > I run a qmail-ldap installation for about 10,000 users. Each has 100MB > > of quota. I use 2 LDAP servers, 2 qmail servers and have all the > > Maildirs stored on a 5.6TB Xserve RAID. >=20 > I would like to ask you if you run your servers on XServe or on FreeBSD. > I'm interested in knowing how you have setup your heterogeneous network > to link FreeBSD to the XServe RAID. On FreeBSD 5.4-STABLE. My FreeBSD (qmail) email servers use NFS. >=20 >=20 > > There are a couple of issues you will run into here. > > > > 1. Mass storage. FreeBSD doesn't support file systems > 2TB, at least > > not that I found decent documentation and support for. >=20 > So how did you go around this limitation? I am buying two Xserves with > XServe RAID and I am positive your experience will help me alot. I've > a great belief that it will.. The 5.6TB XRAID has 2 separate RAID arrays. After RAID-5 overhead it becomes 2.2TB on each array. 2.2TB is 200GB over the max limit, so I had the XRAID slice each array into 2 logical arrays. This gave me 4 x 1.1TB arrays. Each one of those mounted easily in FreeBSD. You may be able use vinum to create them into one large array again, but i didn't bother >=20 >=20 > > 2. Backing up 50,000 Maildirs, where each email is a separate file > > requires something custom. I use Bacula, a network backup tool, and I > > instruct it to do a tar-gzip of each Maildir before backup. This adds > > a bit of overhead, and almost doubles space usage, but it sure beats > > backing up millions of little 4K - 80K files! >=20 > You backup e-mails? That is new to me. How about if the files change as > you run the backup? Oh well. Maildirs don't require file locking. The Maildir may change state but all emails will still be there. Trade off between having OK backups and no backups. FreeBSD's snapshots for large TB volumes isn't stable yet (at least not that I've read) so I don't bother using those. >=20 >=20 > > 3. There is a MAJOR bug with maildirsize, the quota file. These quota > > files go out of sync a lot. From a year of statistics about 0.1% of > > users will likely have out of sync maildirsize files everyday. Who it > > happens to seems to be random. I wrote a custom script that runs every > > 15 minutes to clean up the out of sync maildirsize files. >=20 > This one I am interested in. I experience this alot with Exim. Oh, I use > Exim and not Qmail, but the maildirsize code I believe is from the same > source, no? >=20 I dunno if the code is the same. Good chance if you're seeing maildirsize that are going out of sync for no good reason. >=20 > > Other than those issues my qmail-ldap installation runs super stable. > > On the two mail servers I have serving up IMAP and POP3, their load > > hovers around 0.1 to 0.3 barely anything at all. On my NFS server the > > load is about 0.3... it's barely working too. >=20 > You have separated the servers for IMAP and POP3 or you are > load-balancing? I use DJBDNS and round-robin a domain name with all of my mail servers. It's is very simple to set up and works very well. Load is distributed very evenly. Each mail server provides POP3 and IMAP. Haven't found a need or reason to separate them. Ben.=20 >=20 --=20 blog: http://www.mostlygeek.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?860807bf05050511103bd6412d>