Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 15 Mar 2007 08:28:20 -0700 (PDT)
From:      Mark Crispin <mrc@CAC.Washington.EDU>
To:        Willy Offermans <Willy@Offermans.Rompen.nl>
Cc:        imap-use@u.washington.edu, =?UTF-8?Q?V=C3=A1clav_Haisman?= <v.haisman@sh.cvut.cz>, freebsd-stable@FreeBSD.ORG
Subject:   Re: [Imap-use] Re: imap-uw on FreeBSD 6.2
Message-ID:  <alpine.OSX.0.83.0703150756570.576@pangtzu.panda.com>
In-Reply-To: <20070315120411.GA7953@wiz>
References:  <20070313105155.GA16020@wiz> <45F92E2C.1080406@sh.cvut.cz> <20070315120411.GA7953@wiz>

next in thread | previous in thread | raw e-mail | index | archive | help
  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--0-1687601050-1173972513=:576
Content-Type: TEXT/PLAIN; format=flowed; charset=UTF-8
Content-Transfer-Encoding: QUOTED-PRINTABLE

On Thu, 15 Mar 2007, Willy Offermans wrote:
> On Thu, Mar 15, 2007 at 12:29:48PM +0100, V=C3=A1clav Haisman wrote:
>> I advise you against using IMAP UW, unless you know your mailboxes won't
>> grow beyond just few megabytes or that there will be only few of them.
>> Their reading and rewriting of whenever you make changes can stress
>> server quite a lot.

The problem with taking advice from a sorcerer's apprentice is that often=
=20
the advice is only half-right, or based upon incomplete knowledge.

The large I/O situation described by V=C3=A1clav Haisman is indeed a=20
characteristic of the 1970s vintage traditional UNIX mailbox format which=
=20
is the default mailbox format in UW imapd.

The traditional UNIX mailbox format requires that the program using it=20
read every byte of the mailbox in order to locate the "From " internal=20
header line, and thus to locate and identify messages.

The traditional UNIX mailbox format requires that message metadata as=20
variable-length lines in the message headers.  Alteration of that metadata=
=20
often requires rewriting the remainder of the file as data is slid up or=20
down.  I say "often", since UW imapd insert a modest amount of padding=20
data so that some alterations can be done that only require rewriting the=
=20
message data and nothing else.

Another problem, endemic to all flat file mailbox formats, is that an=20
expunge operation also requires sliding down subsequent messages on top of=
=20
the message being expunged.

However, please note that these are characteristics of a mailbox FORMAT,=20
and not of the IMAP server software.  UW imapd supports several mailbox=20
formats, most notably:
  . The mbx mailbox format is a flat file format with preallocated message
    metadata and message pointers.  The read-every-byte open of traditional
    UNIX mailbox format becomes a per-message seek and 64 byte read in mbx
    format.  mbx was a great choice in the mid 1990s when it was designed,
    but it has gotten long in the tooth.
  . The new mix format is an indexed format in which all the message
    metadata is stored and maintained in a separate index.  Opens are 1
    to 2 orders of magnitude faster than mbx, which in turn is faster than
    traditional UNIX.  Messages are distributed into multiple data files,
    so there are no large files.  Mix is the format that large sites in
    the UW imapd community have either deployed or are moving to deploy.

What this all means is that, yes, switching your IMAP server to another=20
server will probably help your performance -- because that other IMAP=20
server will change your mailbox format to something else!

You can, however, change your mailbox format to something else in UW imapd=
=20
without switching your IMAP server, and achieve the same benefit.=20
Traditional UNIX format is the UW imapd default -- UW imapd pays=20
considerable attention to compatibility and interoperability with existing=
=20
legacy services -- but most UW imapd sites quickly switch formats to a=20
newer technology format.

This is what I meant by "half-right advice and incomplete knowledge."=20
Your performance problem is due to mailbox format, not the server=20
software.  Change your mailbox format, and your problems will be solved=20
(at least until your usage grows to fill the greater capacity provided by=
=20
the new mailbox format...).

If you do decide to switch servers anyway, the two competing servers which=
=20
you should consider are Cyrus and Dovecot.  The developers of those=20
servers are active participants in the IMAP protocol standards community,=
=20
and all of us (UW, Cyrus, Dovecot) talk with each other and go to=20
considerable effort to make our respective implementations interoperable=20
and compliant.  I will leave it to the Cyrus and Dovecot folks to toot=20
their own horns.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
--0-1687601050-1173972513=:576--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.OSX.0.83.0703150756570.576>