Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 15 Dec 2001 22:53:58 -0800
From:      Terry Lambert <tlambert2@mindspring.com>
To:        Anthony Atkielski <anthony@freebie.atkielski.com>
Cc:        Valence Logrus <valence@symboliq.org>, advocacy@FreeBSD.ORG
Subject:   Re: Hotmail _still_ runs FreeBSD!
Message-ID:  <3C1C4506.4343CF9A@mindspring.com>
References:  <20011215041054.H40531-100000@arctic.icelab.net> <030701c18575$2ca87570$0a00000a@atkielski.com> <3C1B7A33.D05FA85A@mindspring.com> <002c01c185f8$82bef1e0$0a00000a@atkielski.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Anthony Atkielski wrote:
> How do you ensure that everything is local delivery if you have multiple
> machines?  With 120 machines and load balancing that can accept a message to
> anyone on any machine, less than 1% of incoming messages will arrive on a
> machine that is also the destination server of the messages.  So 99% of
> messages will have to be rerouted to a different machine.

That's actually not correct.  It depends on your message store
technology.  In any case, the "120 machines" number was based on
forwarding servers.  If the mail was delivered to local disk in
all cases, you could do it with 60.

> And what about users retrieving mail?  How do you make sure that each user
> goes to the correct server?  Either the client must know which server to
> access, or all traffic must be funneled through the same server, although
> you could load balance over those servers.

The easiest way to so this is to use IMAP4 servers for the email
(you need this for multiple mailboxes, in any case, unless you
go to an incredibly custom solution, which is a point of diminishing
returns, in my book).  Then, as part of the LDAP directory entry for
the user record, you include the host(s) on which the storage is to
be considered "local".

You dedicate a number of machines with 4G of RAM, and put an LDAP
directed IMAP4 proxy server on it.  You can easily handle more than
250,000 simultaneous proxy session, assuming you work around the
FreeBSD outbound port hash problem, and assign 4 IP addresses to the
machine internally, to permit 64k*4 outbound interior connections
from the proxy.

Probably, you would need only one or two boxes for this, but let's
say 8, since you may want to include overhead to support stateful
failover.


> > ... the web interface could even contact the
> > remote server directly, should a given email be
> > for one or more recipients constrained to a single
> > target site
> 
> How do you handle relaying restrictions?  If the web machine contacts
> another system directly, unless it has at least an MX record pointing to
> itself--which it cannot have, since it cannot handle incoming mail.
> Unless I'm overlooking something in relaying checks.

You are; but it's not important.  If you are, then maybe the
SPAM'mers are, too, so it's OK.


> > You'd still want a lot of machines to front end
> > the web interfaces, particularly since the CGI's
> > have become incredibly complicated for no good
> > user interface or business reasons (unless you
> > count forcing the use of custom controls "a good
> > business reason").
> 
> The CGIs are complicated because Microsoft is a software company that bloats
> almost everything it produces with lots of additional code.  I've complained
> to them about this before, pointing out that not every single page on every
> single Microsoft site needs to be an ASP, but my complaints fall upon deaf
> ears.  There are lots of people at MS who like to code, and so they write
> code whether it is needed or logical or not.

This isn't the reason; as I stated, you could make a business
case for the complexity.  More than one, actually.


> > FWIW: The FreeBSD servers are pretty much
> > limited to DNS and other infrastructure pieces,
> > rather than customer facing code.
> 
> How did you determine this?  I've never heard FreeBSD being discussed,
> although it would not surprise me if it were used for such functions.

That would be "by talking to Hotmail employees".


> > Actually, Exchange would be fairly easy to fix
> > to handle that type of load, though you would need
> > some cooperation from the NT/2000/XP developement
> > team to optimize a couple of rough corners that are
> > obvious bottlenecks to scalability on that order.
> 
> Last time I heard (about two years ago), Exchange had been scaled to perhaps
> 10 million users in lab experiments, but that is still six times less than
> required for Hotmail.  Exchange is an enormously feature-rich system that
> gobbles resources on a per-account basis.  It also relies far too much on
> monolithic databases that have no integrity features built in, making
> recovery after failure a serious problem as they grow.  I shudder to think
> how long it would take to restore a 10-million-user database in Exchange,
> and unless something changed dramatically in the most recent version,
> restoring the database is the _only_ way to recover from data loss or
> corruption.

Perhaps you missed the part where I said "Hotmail writes incoming
messages to a memory filesystem, not stable storage, for later
processing"?

If a Hotmail server goes down, those messages are lost, utterly.

Realize that Hotmail TOS prohibit business use of Hotmail accounts,
precisely for fear of someone losing an important email.  Hotmail
is not business class email.  The service I've described, is.


> > Surprising as this is... Microsoft has Microsoft
> > source code.
> 
> Surprising as this is, Microsoft can't always afford to modify its own code.
> Building a Hotmail-specific version of Exchange is not economically
> feasible.  If it doesn't run with the stock version of Exchange (and
> apparently attempts to make it do so in the past have failed), then it can't
> run on Exchange at all.

You could do this trivially, with a tiny policy decision, and throwing
hardware at the problem.  I will not describe the approach further, for
fear they would implement it.  If they want to do it, they will have to
pay me a lot for the information.


> > I rather expect that the main reason they have
> > not done more to get away from FreeBSD/Solaris
> > is a resource/reward issue.
> 
> Maybe, although they should be able to make anything work if they throw
> enough hardware at it.  If they wanted to force use of Exchange for
> political reasons, and if the product scaled acceptably, they could just add
> machines until it worked.  Certainly Exchange architecture theoretically
> facilitates this, but perhaps it doesn't work well beyond a certain point in
> practice (I don't recall exactly _why_ Exchange didn't work for Hotmail).

It was scaling; but they weren't clever enough in their approach;
I could have made it work (I tend to solve problems "differently").


> In any case, not using Exchange for Hotmail is not really a bad reflection
> on Microsoft; it is such a specific and unusual mail application that it
> would be unreasonable to expect anything to scale to handle it out of the
> box.

This is irrelevent, since they decided to convert, and the
conversion failed.  If they had taken that tack in the first place,
then it would be an acceptable brush-off.

I expect that they learned an incredible amount in the process, and
are going to be able to do it eventually, even if not now.


> > If I could do the migration to Windows, then they could
> > do the migration to Windows...
> 
> When was the last time you migrated a 60-million-user e-mail system to
> Windows?

I built out a 500,000 user mail system for IBM Web Connections in
Rochester, NY, and it could have easily scaled to more users simply
by throwing hardware at it and loading my distribution CDROMs based
on the role you wanted the new machines to take.  Does that count?
It ran on 6 machines, but they were RS/6000's with crossbar busses
with the processors running at a whopping 166MHz (the crappy state
of PC hardware is why so many machines would be required; I did
assume SCSI, however, for my estimated numbers of machines).

-- Terry

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-advocacy" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3C1C4506.4343CF9A>