High Availability Distributed Solution On The AXIGEN Mail Server
High Availability Distributed Solution On The AXIGEN Mail Server
This document describes an implementation of a large-scale messaging solution relying on the AXIGEN Mail Server software. The global architecture for the solution is described along with implementation details and operation and maintenance procedures.
1.2 Intended Audience
The information in this document is intended for users who are evaluating the benefits of a distributed, high availability solution as well as for integrators and operational personnel. The components of such a solution, both software and hardware, are also listed in this document thus ensuring the ability to assess overall associated costs.
1.3 Definitions, Terms and Abbreviations
• Vertical scalability – potential increase in processing capacity of a machine attainable by hardware upgrades;
2.1.1 Statefull Services
Non-distributed email solutions, where account information (configuration and messages) is stored on a single machine allow vertical scalability through hardware upgrades (CPU, RAM, disk). However, due to limitations in a typical machine (i.e. max 2 CPU, max 4 GB RAM etc) an upper limit is eventually reached where one can no longer upgrade one machine – we shall refer to this as vertical scalability limit.
When the vertical scalability limit is reached, the only solution available is to distribute account information (configuration and mailbox) on more than one machine – we shall refer to this as horizontal scalability. Since information for one account is atomic and cannot be spread across more machines, the solution is to distribute accounts on more than one machine. This way, for a single account, there will be one machine responding to requests (IMAP, POP, SMTP) for that specific account. Thus, when the overall capacity (in terms of active accounts) of the messaging solution is reached, adding one more machine to the solution and making sure new accounts are created provides a capacity upgrade, therefore allowing virtually unlimited horizontal scalability.
It must be noted that, since each account of the system is serviced by a specific node, a centralized location directory must be available to provide location services. In our case an LDAP system will store information about which node is able to service requests for a specific account.
2.1.2 Stateless Services
Since stateless services do not store information over multiple sessions, we can assume that two different machines are able to service requests for the same account. This way, horizontal scalability can be achieved by simply adding more machines providing the same service in the exact same configuration.
The only remaining requirement is to ensure that requests to a specific service are distributed evenly throughout the machines providing that specific service (i.e. if the system contains two machines providing IMAP proxy services, half of the incoming IMAP connections must reach one of the machines and the rest of the connections must reach the other machine). This functionality is provided by a load balancer, be it hardware (dedicated) or software (Linux machine running LVS).
2.2 High Availability and Fault Tolerance
2.2.1 Statefull Services
Consider the fact that, for statefull services, requests for one specific account are made to a specific machine. If that specific machine experiences a fault and can no longer respond to requests, none of the other machines are able to service the account in question. A mechanism is required to ensure that, in the event of a catastrophic failure on one machine, some other node must take-over the task of servicing requests for that account thus providing high-availability.
RedHat Clustering Suit provides this exact functionality; it ensures that, if one node running a statefull service fails, another node will automatically detect the fault and start the required service in-place of the failed node, providing minimal downtime to that service.
2.2.2 Stateless Services
In the case of stateless services, since any of the nodes providing the same service is able to respond to requests for any account, the only requirement is to make sure that the request distribution mechanism (load balancer) can detect when one of the nodes no longer responds to requests and ceases to direct service requests to that specific node. The total request processing capacity is decreased (the system will respond slower, since one node no longer processes requests), but all service requests can still be processed.
3 Solution Architecture
The solution uses three tiers to provide the required functionality. The load balancer tier provides services for network layer 4 (transport), TCP connections, and is completely unaware of account information; it only provides distribution of connections to the nodes in the front-end tier.
The front-end tier comprises of nodes running proxy services and SMTP routing services. Its task is to ensure that messages and connections are routed to the appropriate node (depending on the account for which a request is being performed) in the backend tier.
Finally, the backend tier provides access to persistent data (such as account configuration and mailbox data); each node in the backend tier is capable of responding to requests for a set of accounts. No node in the backend tier is capable of servicing requests for an account that is serviced by a different node.
3.2 Service-level High Availability
Depending on the service type (statefull or stateless), high availability is achieved differently.
3.2.1 High Availability for Statefull Services
We shall define, for the backend tier, the following terms: service-instance and cluster-node. A service instance comprises of a running service and storage for a specific set of accounts. A cluster node is a machine. Each machine is capable of running one (typically) or more service instances.
High-availability in this tier is achieved by making sure that, in the event of failure of a physical node, the service instance is automatically started on a different cluster node. Currently, there are a number of software packages providing this functionality; AXIGEN was tested with RedHat Clustering Suite.
A disadvantage of this high-availability mechanism is the delay that is induced between the time a service on one node fails (or the node fails completely) and the time the service is restarted on a different node. This delay is caused by the time required for the cluster software (RHCS, in our case) to detect the failure and the time required to remount the data partitions on a different node, activate the virtual network interfaces and start the service. During this period (which may vary from 10 seconds to a couple of minutes) the service is not available for the users.
3.2.2 High Availability for Stateless Services
In the case of stateless services (services in the front-end tier), the load distribution system also provides high availability capabilities. The load balancer automatically distributes requests (at TCP level – network layer 4) to all the nodes in the front-end tier, based on the configured algorithm. If one node in the front-end tier fails, the load balancer will automatically detect the failed node and will no longer distribute connections to it. A major advantage over the statefull services high availability mechanism is that, due to its ‘active-active’ nature, a node failure causes no service downtime.
3.3 I/O High Availability
The high-availability used at service level provides a full ‘no-single-point-of-failure’. However, any faulty hardware component in a node causes that node to be unusable thus diminishing the total processing power (hence the total transaction capacity) the solution provides. There is a mechanism which makes sure that, even in the case of an I/O controller failure, the node can continue to provide the service; this relies on having duplicate I/O controllers on each node and a software method of failover (rerouting I/O traffic from the faulty controller to the healthy one).
The I/O high availability can be used for disk I/O and network I/O fault tolerance, provided that duplicate controllers are available on the nodes. This reduces the occurrence of service downtime in the case of statefull services; if an I/O controller fails, the service would need to be restarted on a different node.
Read more on the AXIGEN site.