Emails in Numbers

The article analyzes the results of independent tests performed by PC Magazine Romania comparing the AXIGEN Mail Server against two open source alternatives, Sendmail (with Dovecot) and Postfix (with Cyrus):

We thought of conducting a comparative performance study for three solutions that cover the four basic functions of electronic messaging: message receiving, their delivery to the user’s mailboxes, message storage and user accessing stored emails. Two usage scenarios were considered: business and ISP.

The three tested solutions are:

• Sendmail (message receiving and delivery) + Dovecot (message storage and access)
• Postfix (message receiving and delivery) + Cyrus (message storage and access)
• AXIGEN (complete solution)

The tests consisted in sending messages with a predetermined size to the servers and checking their acceptance in the users’ mailboxes.

The large number of spam messages from the total traffic of received email messages (estimated by Radicati, in 2007, at 72% of all traffic) generates frequent periods of intensive server usage. To verify the servers’ ability to respond in overload conditions, their response time to requests on 1, 2, 4 and 8 parallel connections was tested.

Business type scenarios (medium/large companies)

Characteristics
Medium and large sized companies generally employ their own messaging solutions for security and efficiency reasons. Typically, messages sent in the business environment are medium sized (13.6kB) and the employees connect to an e-mail client (for example: MS Outlook, Mozilla Thunderbird etc.) trough the IMAP protocol.

Testing results
The most relevant performance indicators of a messaging solution are the number of accepted messages by the server and the number of delivered messages to the user’s mailbox within a time unit. Ideally, the two indicators are equal; therefore, the server is able to immediately deliver all received messages.

After running the tests
We noticed that, in the case of Sendmail, the accepted number of messages is almost equal to the delivered number of messages witch ensures the server’s reliability; also, the number of these messages increases when the number of parallel connections is increased. However, from 4 to 8 parallel connections, there’s only a minor increase of the received/delivered message number, leading us to conclude that the maximum performance level is archived; no matter how many parallel connections are added, the total performance doesn’t increase anymore.

We notice that, for Postfix, once we increase the number of parallel connections, the difference between the number of accepted and delivered messages is significantly greater, the latter being much smaller. It can be concluded that a big part of the processing power is used to accept messages; sadly, the delivery is affected by this behavior, the immediate effect being the constant message queue increase, which finally causes a server blockage. Compared to Sendmail, Postfix works a little better in the case of a single connection, but the performance balance switches in favor of Sendmail as the number of parallel connections is increased.

Unlike the first two solutions, AXIGEN maintains a balance between the number of received and delivered messages and its performance highly increases when new parallel connections are being added, working almost 13 times better than Sendmail and Postfix at 8 parallel connections.

To conclude, even if in normal traffic situations a 7-20 messages/second performance is satisfactory and the Sendmail or Postfix solutions behave acceptably, during peak traffic periods, such as virus outbreak situations, spam attacks, when sending large numbers of messages (e.g emails to large distribution lists) or in case of server attacks, AXIGEN proves to be much more reliable.

For the article in full, including the ISP scenario and graphical representations of conducted tests, please visit:
Emails in Numbers