An Ultra-HA, full Mult-Master E-mail cluster with iRedMail,

djbon2112 Sep 20th, 2014 (edited) 4,975 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. An Ultra-HA, full Mult-Master E-mail cluster with iRedMail, MariaDB, and IPVS
  2. by Joshua Boniface
  4. -----
  6. **
  7. 0. Edits
  8. **
  10. 2015-05-11:
  11. As I made my cluster more expansive, I ran into an interesting problem that is caused by the simplicity of the setup in this guide, and was hidden from me because I was actually using a more complex setup. It concerns the ldirectord VIPs for accessing the database servers. I do apologize if you ran into any frustration due to this, as it drove me fairly mad when I ran into it myself. The changes are in the relevant section on ldirectord and minor changes to the cluster layout.
  13. **
  14. 1. Introduction:
  15. **
  17.         iRedMail is a very nifty piece of software. Setting up a full mail server on modern Linux is indeed possible; there are guides for every part of the system, preconfigured templates, and many-a mailing list post. However, iRedMail does something special: it makes it easy. Easy to install, easy to administer, and and easy to use. However, there are very few guides on how to deploy a *complete*, clustered iRedMail solution. Let's talk a bit about what I mean by that.
  19.         I know e-mail, having deployed Debian-Linux-based carrier-grade mail platforms as part of my job. Setting up a cluster for production use, you want something that's fault-tolerant on every level. If you have 3 physical servers, you want to make sure you can keep your systems running even if you have to bring one down for maintenance, or due to a catastrophic hardware fault. You want to make sure you don't lose anything because one server crashed. However, many HA setups are not full multi-master; sure, you can lose a node, but it better not be server*1*. The setup I propose below addresses this: it doesn't matter which of your N nodes fails, you can always perform every task you need; and without proper monitoring, you might not even notice you have a problem! Some may think this is overkill, but the result is very compelling for anyone who values a 100% uptime!
  21.         The setup I'm using is based on Debian Wheezy (7), the latest stable release of the Debian GNU/Linux operating system; 64-bit is of course recommended. I am very dilligent on separating services, for good reason: ease of manageability and flexibility, and as a result this guide uses slightly more VMs and IP addresses than one may expect. I will break down the cluster now so you have a better idea of how this is running.
  23.         Please note that while this guide has parts that can be copy-pasted, and most specified shell commands and config files will work as intended, I do expect you to understand what you're doing, and RTFM if you don't; a decent knowledge of Linux System Administration is a must here. I don't go into any detail about creating your VMs, or any basic system administration tasks or commands, or what specific config options in files do. Also, all IP addresses/hostnames are fictitious and must be replaced, and anything in <> square brackets must be filled in with you own information. Finally, please note that I offer NO GUARANTEED SUPPORT FOR THIS GUIDE, though if you have a good question I'll probably answer it.
  26. **
  27. 2. The Cluster:
  28. **
  30.         My home cluster is a fairly simple beast: there are two distinct hypervisors ( and running KVM, and a single file server ( At this time, mostly due to budget reasons (it's a homelab, and those cost a lot of money in power!), I am not replicating to a second fileserver, and hence the backend Maildir storage is not HA in my setup. This can be acomplished in a huge number of ways (glusterFS, DRBD, manual sync) but is outside the scope of this guide. I assume that "" is some device, providing a single NFS interface for backend storage of Maildirs.
  32.         The Virtual Machines running on and are served via NFS from filer1, as are the Maildirs used for storing e-mail. This NFS is on a private network local to the servers, and this network also carries LDAP sync and Database traffic. The virtual machines are tied to a hypervisor: each *1* server is on and each *2* server is on It's worth pointing out now that this cluster could easily be expanded to 3 hypervisors (and hence a *3* server for each service) if desired; this is recommended for the Database cluster in particular, however in my setup the is the third, quorum-holding database server.
  34.         I expect your setup to be slightly different. If so, just adapt this guide; I use consistent naming throughout (sed might be your friend here)!
  37. 2a) Virtual machines and networking
  39.         The cluster comprises the following service VMs, all running Debian Wheezy:
  41.           i) lbX, IPVS load balancers
  42.          ii) dbX, MariaDB/Galera MySQL servers
  43.         iii) mailX, iRedMail/LDAP servers
  45.         Additionally, two VIP addresses for the load balancers is required:
  47.          iv) lb0a, IPVS VIP for external mail services
  48.           v) lb0b, IPVS VIP for databases
  50.         And one IP for the file server containing Maildirs:
  52.          vi) filer1, NFS server
  54.         The entire cluster therefore uses 9 IP addresses (ignoring the hypervisors, and any other VMs you might have set up). For the purposes of this guide, I assume two networks: "public",, and "private" , You can omit either network and use a single private network; my proper "public" network uses routable public IPs, while the "private" network is unrouted, and certain replication traffic and NFS are kept on the "private" network for security. You can ignore this convention if you want, or even use Masquerade mode with IPVS, to hide all these services behind a single "public" IP and keep it all behind NAT. Whatever works for your environment! If you don't have a proper DNS setup, you can use this template in your "/etc/hosts" file on each host.
  56.         # "Public"
  58. # (VIP)
  59. # (VIP)
  66.         # "Private"
  67.       filer1.local
  68.       lb1.local
  69.       lb2.local
  70.       db1.local
  71.       db2.local
  72.       mail1.local
  73.       mail2.local
  75.         Note that there are no VIP in the "private" network: since all its services are load-balanced from the "public" IP, it is unnecessary for there to be a "local" VIP. I recommend firewalling the second VIP address to block MySQL traffic from the outside world if you are using a true public IP, since the MySQL cluster should not be visible to the outside world.
  78. 2b) A note on example conventions
  80. Thoughout this guide, when command-lines are given, the following rules will be held:
  82.   i) the beginning of the prompt will indicate the server name, either as:
  83.         server1 # <command>
  84.      for a specific server ID, or:
  85.         serverX # <command>
  86.      for all servers in that category, or even:
  87.         serva1, servb2 # <command>
  88.      for two specific server names
  89.  ii) the seperator character shall be isolated on both sides by a space (for ease of copying a single command, but
  90.      discouraging block copying) and will consist of:
  91.         # - for a root-level account
  92.         $ - for an unprivileged account
  93. iii) for simplicity, most commands in this guide are written as an unprivileged user with "sudo" prepended;
  94.      commands that require the actual *root* account (e.g. the setup script) will use # instead
  95.  iv) when editing a text file, the raw contents from server1 will be presented after the command (usually 'sudo
  96.      vim'), followed by a 'diff' of the differences between server1 and server2, if necessary; one can extrapolate
  97.      the third server or any other differences if desired
  98.   v) any comments regarding a text file will follow the output and diff, prepended by a [*] for each comment
  101. **
  102. 3. Setting up IPVS, ldirectord, and keepalived
  103. **
  105. Chapter source:
  107.         The first and probably easiest part of this cluster is the load balancing configuration. It is a very straightforward setup, with 2 load-balancers sharing 2 VIP addresses: if contact is lost, keepalived moves the VIPs between the two servers on a weighted basis (lb1 is prefered to lb2).
  109. Start by installing the required packages on both hosts.
  111. lbX $ sudo apt-get update
  112. lbX $ sudo apt-get upgrade
  113. lbX $ sudo apt-get install ipvsadm ldirectord keepalived
  116. 3a) keepalived
  118.         Begin by editing the keepalivd configuration. This will set up the VIPs between the two load balancers, and allow them to fail over from lb1 to lb2 in the event lb1 goes down, thus preserving services to anyone connecting to the cluster through this IP.
  120. lbX $ sudo vim /etc/keepalived/keepalived.conf
  122. vrrp_instance VI_1 {
  123.         state MASTER
  124.         interface eth0
  125.         virtual_router_id 1
  126.         priority 200
  127.         authentication {
  128.                 auth_type PASS
  129.                 auth_pass mySuperSecretPassw0rd
  130.         }
  131.         virtual_ipaddress {
  132.       ;
  133.       ;
  134.         }
  135. }
  137. 5c5
  138. <         priority 200
  139. ---
  140. >         priority 100
  142. [*] The adjusted priority on lb2 allows lb1 to take precidence and prevent flapping between the two load balancers. If you have a third lbX host, you can make its priority something less than 100 to ensure it will be last in the chain.
  144. Restart the keepalived service on both hosts:
  146. lbX $ sudo service keepalived restart
  148. You should now see the VIPs in the list of IP addresses on lb1:
  150. lb1 $ ip a
  151. [...]
  152. eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
  153.     link/ether 11:11:11:11:11:11 brd ff:ff:ff:ff:ff:ff
  154.     inet brd scope global eth0
  155.     inet scope global secondary eth0
  156.     inet scope global secondary eth0
  157. [...]
  159. Now stop keepalived on lb1, and observe lb2: the IP addresses will transfer after a second or two:
  161. lb1 $ sudo service keepalived stop
  163. lb2 $ ip a
  164. [...]
  165. eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
  166.     link/ether 11:11:11:22:22:22 brd ff:ff:ff:ff:ff:ff
  167.     inet brd scope global eth0
  168.     inet scope global secondary eth0
  169.     inet scope global secondary eth0
  170. [...]
  172. You can then restart the keepalived service on lb1 and the IP addresses will return to it, exactly as expected.
  175. 3b) ldirectord
  177.         Next is the ldirectord configuration. ldirectord is a load balancer using IPVS, the Linux kernel virtual server, to distribute traffing entering on a VIP between a number of "real servers", . It contains a number of different load-balancing and routing options, however for our purposes, with a "public" network, we will use the 'routed' mode, whereby traffic is directly routed from the VIP to the real server, which has configured on a loopback interface the VIP, allowing traffic to be sent directly back to the client from the real server, reducing load on the load balancers. In effect, the routers simply keep track of incoming packets while the outgoing packets flow right to the client.
  179. ldirectord works by performing regular health checks on the real servers; if one is found to be non-working, it is removed from the IPVS configuration, thus preventing clients from being directed to a dead server. Once the service has been restored, ldirectord re-adds the real server to the IPVS configuration, and load-balancing resumes.
  181. The following file contains all the services that will be provided in HA mode for client access, the list of which is: MySQL, HTTP/S, IMAPS, POPS, and SMTPSUB. I don't allow unsecured access via IMAP or POP3 directly to my mail servers, but you can add these services if desired.
  183. lbX $ sudo vim /etc/
  185. logfile="daemon"
  186. fallbackcommand=""
  187. failurecount=3
  188. checkinterval=5
  189. fork=yes
  191. # MySQL database to db1/db2
  192. virtual=
  193.         real= gate
  194.         real= gate
  195.         service=mysql
  196.         scheduler=sh
  197.         login="monitor"
  198.         passwd="monitoringPassw0rd"
  199.         request="SELECT * from monitoring.monitoring;"
  200. # Mail services to mail1/mail2
  201. virtual=
  202.         real= gate
  203.         real= gate
  204.         service=http
  205.         scheduler=sh
  206.         request="ldirectord.txt"
  207.         receive="ldirectord"
  208. virtual=
  209.         real= gate
  210.         real= gate
  211.         service=https
  212.         scheduler=sh
  213.         request="ldirectord.txt"
  214.         receive="ldirectord"
  215. virtual=
  216.         real= gate
  217.         real= gate
  218.         service=imaps
  219. virtual=
  220.         real= gate
  221.         real= gate
  222.         service=pops
  223. virtual=
  224.         real= gate
  225.         real= gate
  226.         service=smtp
  228. [*] Both servers are identical.
  229. [*] Service checks for MySQL and HTTP/S will be addressed in their relevant sections.
  231. Reload the ldirectord service, and use "ipvsadm" to view the resulting IPVS configuration (IP-to-hostname translation is used, if you don't have reverse DNS configured you will see IP addresses):
  233. lbX $ sudo service ldirectord restart
  234. lbX $ sudo ipvsadm
  235. IP Virtual Server version 1.2.1 (size=4096)
  236. Prot LocalAddress:Port Scheduler Flags
  237.   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  238. TCP sh
  239.   ->        Route   1      0          0        
  240.   ->        Route   1      0          0
  241. TCP sh
  242.   ->       Route   1      0          0        
  243.   ->       Route   1      0          0
  244. TCP sh
  245.   ->      Route   1      0          0        
  246.   ->      Route   1      0          0
  247. TCP wrr
  248.   ->      Route   1      0          0        
  249.   ->      Route   1      0          0
  250. TCP wrr
  251.   ->       Route   1      0          0        
  252.   ->       Route   1      0          0
  253. TCP wrr
  254.   -> Route   1      0          0        
  255.   -> Route   1      0          0
  257. However, since you have not yet configured any services, there will be no real servers in your output, only the lines containing "":
  259. IP Virtual Server version 1.2.1 (size=4096)
  260. Prot LocalAddress:Port Scheduler Flags
  261.   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  262. TCP sh
  263. TCP sh
  264. TCP sh
  265. TCP wrr
  266. TCP wrr
  267. TCP wrr
  269. Once this guide is done, compare the resulting output from "ipvsadm" to the above output, and you should see it match!
  271. This concludes the configuration required on the load balancers themselves. However, one more piece of configuration must be done to *each* real server: it must have the VIP address for the host added to a loopback interface to allow services on the server to use that address to reply to clients. This is a required part of the "direct-routing" mode used in IPVS. If you are using an alternate routing mode (for example Masquerade), you do not need this step. On each dbX and mailX host:
  273. serverX $ sudo vim /etc/network/interfaces
  275. # Loopback interface
  276. auto lo
  277. iface lo inet loopback
  279. # IPVS-DR loopback interface
  280. auto lo:0
  281. iface lo:0 inet static
  282.         address
  283.         netmask
  284.         pre-up sysctl -w net.ipv4.conf.all.arp_ignore=1
  285.         pre-up sysctl -w net.ipv4.conf.all.arp_announce=2
  287. # Other interfaces, server-specific...
  288. [...]
  290. [*] All servers are identical, except dbX which use as the IP.
  292. This concludes the configuration of the load balancer setup, and the VIP that will direct requests to the client machines.
  294. ---
  295. EDIT 2015-05-11: WHY DO I NEED TWO VIPs?
  297. The simple reason is: because of how networking works. In the original version of this guide, I recommended setting this up with a single VIP: that VIP would be present on four servers: db1, db2, mail1, and mail2. The idea was that outgoing database requests from mailX would hit the VIP on lb0, then be redirected to one of the dbX hosts, like so:
  299. mail1 -> lb0 -> db?
  301. However, by having the *same* VIP address on the mailX hosts that we do on the dbX hosts, we run into a major problem: when mail1 tries to talk to lb0, what actually happens is it tries to talk to itself, on it's lo:0 interface! And naturally, since MySQL isn't running on mail1 (or mail2), it will fail to connect!
  303. There are a number of ways to solve this problem, but truthfully to preserver the existing load balancing setup, the simplest is to just add a second VIP dedicated to the database cluster. You can then alias "" to "lb0b" and continue with the rest of the guide unmodified.
  304. ---
  306. **
  307. 4. Setting up MariaDB and Galera multi-master SQL
  308. **
  310. Chapter source:
  312.         As seen above, one of the load-balanced services is MySQL. Databases are used extensively in e-mail servers: they hold information about active accounts, sessions, filter policies; the list goes on. The services of the dbX servers could be integrated into the mailX servers themselves, however in my usage it makes more sense to separate them. You can easily run all of the following on the mailX servers and reduce your IP usage by two if you so desire (just don't forget to edit the file in Chapter 3 to match!)
  314. The MySQL cluster will be provided by MariaDB, a community-driven fork of Oracle's MySQL, and headed by the original developers of MySQL. It is combined with the Galera replication engine to allow a multi-master cluster than can be load-balanced by IPVS. I am using version 5.5 for maximum compatibility, though the newer releases could be used as well. To prevent split-brain, we also use a third host in the Galera cluster, which will be provided by the filer1 server; if you are using this guide to set up a 3-server cluster, you can exclude that host as quorum will be provided by 3 dbX servers. Run all commands below on filer1 as well as dbX.
  316. Start by adding the MariaDB sources (other mirrors can be found at into your apt configuration:
  318. dbX $ sudo apt-get install python-software-properties
  319. dbX $ sudo apt-key adv --recv-keys --keyserver 0xcbcb082a1bb943db
  320. dbX $ sudo add-apt-repository 'deb wheezy main'
  322. Then update and install the required packages:
  324. dbX $ sudo apt-get update
  325. dbX $ sudo apt-get install rsync galera mariadb-galera-server
  327. You will be asked for a root password during setup; ensure it is identical on all hosts. Once installed, stop the mysql process as we need it off for the next steps.
  329. dbX $ sudo service mysql stop
  331. The Galera configuration below is extremely simple; read the Galera documentation for more advanced configuration options. Set the local IP addresses of the cluster members in the "wsrep_cluster_address" line, to keep replication traffic on the unrouted local network. You can also set the "wsrep_cluster_name" to a new value; this is in effect a shared secret for the cluster.
  333. dbX $ sudo vim /etc/mysql/conf.d/galera.cnf
  335. [mysqld]
  336. # MySQL settings
  337. binlog_format=ROW
  338. default-storage-engine=innodb
  339. innodb_autoinc_lock_mode=2
  340. query_cache_size=0
  341. query_cache_type=0
  342. bind-address=
  343. # Galera settings
  344. wsrep_provider=/usr/lib/galera/
  345. wsrep_cluster_name="db_cluster"
  346. wsrep_cluster_address="gcomm://,,"
  347. wsrep_sst_method=rsync
  349. [*] All servers are identical.
  351. Also clone the /etc/mysql/debian.cnf file from db1 to the other database hosts; this will, combined with the tweaks below, prevent "debian-sys-maint" access denied warnings when starting the other cluster nodes.
  353. Warning: Before we continue, I have discovered a bug in this setup. Because of the IPVS-DR loopback, the Galera cluster will sometimes fail to start on the second or third node of the cluster. The reasons I do not completely understand. To mitigate this however, I made a modification to the "/etc/init.d/mysql" initscript to add an "ifdown lo:0" and corresponding "ifup lo:0" at the beginning and end, respectively, of the "start" function. I recommend doing this to save you hours of headaches!
  355. Once the configuration is in place on all nodes, we can start the cluster on the first node:
  357. db1 $ sudo service mysql start --wsrep-new-cluster
  359. The "--wsrep-new-cluster" directive creates a new active cluster; if all hosts in the Galera cluster go down, you will need to execute this command on a node again to start up the cluster. Data is of course preserved when running this command, and the host it is run on will become the "primary" sync source for the other members of the cluster.
  361. On the remaining nodes, start the MySQL service normally:
  363. db2,filer1 $ sudo service mysql start
  365. If all goes well, they will connect to the cluster master, and data will synchronize. Check the number of nodes in the cluster with:
  367. db1 $ mysql -u root -p -e 'SELECT VARIABLE_VALUE as "cluster size" FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME="wsrep_cluster_size"'
  368. +--------------+
  369. | cluster size |
  370. +--------------+
  371. | 3            |
  372. +--------------+
  374. There are a number of configuration tweaks that must be performed to properly use the MySQL cluster as expected. Enter the database server, and:
  376. db1 $ mysql -u root -p
  377. Enter password:
  378. Welcome to the MariaDB monitor.  Commands end with ; or \g.
  379. Your MariaDB connection id is 104893
  380. Server version: 5.5.38-MariaDB-1~wheezy-wsrep-log binary distribution, wsrep_25.10.r3997
  381. Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
  382. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
  383. MariaDB [(none)]>
  385.   i) Create a "new" root user with global host access. Yes, you can restrict this per-host, but that gets messy fast
  386.      and if you have a *properly secured* network, you shouldn't have to worry about this too much.
  388. MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION IDENTIFIED BY '<MySQL root password>';
  389. MariaDB [(none)]> GRANT PROXY ON ''@'' TO 'root'@'%' WITH GRANT OPTION;
  391.      Test that this user works by logging in to another MySQL shell, and if it works fine, drop all "old" root users:
  393. MariaDB [(none)]> SELECT User,Host from mysql.user;
  394. [view the resulting list of users]
  395. MariaDB [(none)]> DROP USER 'root'@'db1';
  396. MariaDB [(none)]> DROP USER 'root'@'db2';
  397. [etc.]
  399.  ii) Create a "new" debian-sys-maint user, with slighly more restricted access than the root user; again this user
  400.      should be for the global host for simplicity.
  404.      Make sure that the "/etc/mysql.debian.cnf" file is identical between all the dbX nodes. And like the root user, drop all "old" debian-sys-maint users:
  406. MariaDB [(none)]> DROP USER 'debian-sys-maint'@'db1';
  407. MariaDB [(none)]> DROP USER 'debian-sys-maint'@'db2';
  408. [etc.]
  410. Attempt to stop the mysql service on any one node; it should suceed without any warnings or errors about permission denied; restart it and resume configuration.
  412. Now we will add some data to the cluster and observe its replication. The data used is, conveniently, the monitoring framework required by ldirectord.
  414. db1 $ mysql -u root -p
  415. MariaDB [(none)]>
  417. Begin by creating a new database called 'monitoring'; these values were set in Chapter 3, in
  419. MariaDB [(none)]> CREATE DATABASE monitoring;
  421. Create a new user, 'monitor', identified by the password 'monitoringPassw0rd', and grant select access to the 'monitoring' database:
  423. MariaDB [(none)]> GRANT SELECT ON monitoring.* TO 'monitor'@'%' IDENTIFIED BY 'monitoringPassw0rd';
  425. Now, change into the monitoring database, and create a table called "monitoring" containing some data:
  427. MariaDB [(none)]> USE monitoring;        
  428. MariaDB [monitoring]> CREATE TABLE monitoring (data VARCHAR(1));
  429. MariaDB [monitoring]> INSERT INTO monitoring (data) VALUES ("X");
  430. MariaDB [monitoring]> SELECT * FROM monitoring.monitoring;
  431. +------+
  432. | data |
  433. +------+
  434. | X    |
  435. +------+
  436. MariaDB [monitoring]> quit
  437. Bye
  439. You have now set up the monitoring table that the ldirectord daemon will connect to and attempt to judge your hosts' health. If everything is configured and working right, you should now see the real servers in the output of "ipvsadm" on lbX:
  441. lbX $ sudo ipvsadm
  442. IP Virtual Server version 1.2.1 (size=4096)
  443. Prot LocalAddress:Port Scheduler Flags
  444.   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  445. TCP wrr
  446.   ->        Route   1      0          0        
  447.   ->        Route   1      0          0
  448. [...]
  450. You can see the data replication by checking any other node: you should be able to see the monitoring database and its table. But if both your dbX servers are in the IPVS configuration above, you should be good to go! Now you can access the databases, load-balanced between the two dbX VMs, from the VIP address at I recommend aliasing/pointing "" to "" now as this will be used as a referenced name later.
  452. A note about filer1: When setting up the DB, we used filer1 as our third, quorum-holding host. However, as that is not a proper "database" server, it is NOT added to the load-balanced cluster. It is, in effect, a write-only copy of the data to preserve quorum. If you are using three physical servers, and hence three dbX servers, you should be able to use just those 3 to maintain quorum, and load-balance over all 3, in which case you can avoid putting any MySQL on the filer1 server.
  454. This concludes the configuration of the database cluster, which is now ready for data to be added from the mailX servers, via the load-balanced VIP address at
  457. **
  458. 5. Highly-available, LDAP-backed iRedMail e-mail
  459. **
  461. Chapter source: iRedAdmin Wiki/Existing Tutorials
  463.         And now for the Pièce de résistance, the whole reason for this tutorial: the HA iRedMail cluster! For this setup, we will be using a multi-master LDAP backend to store user data; each mail server will connect to its own, local, LDAP database and those databases will be synchronized between the two servers. This allows iRedAdmin management, as well as user access, to be passed through the load balancer: if a mail server goes down, regardless of which one, the mail administrator(s) can still make changes and continue working as if nothing happened to the infrastructure backend; let the sysadmins worry about that!
  465. Note that mirrormode LDAP multi-master clusters are ideally used with a "preferred" master. For this reason, I don't recommend *actually* using the VIP address to access iRedAdmin under normal situations: manage from mail1, and if needed, manage from mail2 if mail1 is down. This helps preserve consistency so that you can trust one particular host if an LDAP split-brain happens.
  468. 5a) iRedMail installation with the HA MySQL backend
  470. To begin, become root on both mail servers:
  472. mailX $ sudo su
  473. mailX #
  475. Now download and extract the latest version of iRedMail, at the time of writing iRedMail-0.8.7:
  477. mailX # wget iRedMail-0.8.7.tar.bz2
  478. mailX # tar -xvjf iRedMail-0.8.7.tar.bz2
  480. Before we begin installing iRedMail, add a fully-privileged user to the database server that can be used by the install script to set up the databases. This user can be removed after installation; alternatively, you could use root but this is not recommended.
  482. db1 # mysql -u root -p
  483. MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'admin_iredmail'@'%' WITH GRANT OPTION IDENTIFIED BY '<password>';
  485. Next, mount the NFS Maildir storage from filer1 on both mailX hosts (I prefer the default "/var/vmail"); ensure you add it to "/etc/fstab" as well:
  487. mailX # mkdir /var/vmail
  488. mailX # chattr +i /var/vmail
  489. mailX # mount -t nfs -o vers=3,noatime,nodiratime,noexec filer1.local:/srv/var/vmail /var/vmail
  490. mailX # echo "filer1.local:/srv/var/vmail /var/vmail nfs vers=3,noatime,nodiratime,noexec 0 0" >> /etc/fstab
  492. Configuration should begin on mail1 first: this will allow us to generate an iRedMail installer 'config' file, which we will then use to ensure mail2 is configured with the same settings.
  494. Start the iRedMail installer with variables specifying the database host, user, and grant host (in our case, '%' for simplicity in our MySQL users):
  496. mail1 # cd iRedMail-0.8.7/
  497. mail1 # MYSQL_SERVER='' MYSQL_ROOT_USER='admin_iredmail' MYSQL_GRANT_HOST='%' bash
  499. Follow the directions as per the standard iRedMail setup procedure. In particular, choose an LDAP backend, and choose the NFS directory for the Maildir storage. Also ensure that you save any password you entered: these will eventually be the cluster master passwords. During setup, you will be asked for the password for "admin_iredmail" you set above in order for the installer to access the MySQL cluster. Also, don't use an additional domain when asked for your first virtual domain: use "". This will simplify our deployment and allow you to add actual domains to the full cluster later.
  501. Once the iRedMail setup completes, your first node will be operational! Feel free to test it out, and inspect the database servers to confirm that the data for the iRedMail server was properly added to the MySQL cluster backend and is replicating between the hosts as expected.
  503. Next, copy the "config" file from the iRedMail installer directory over to the second server. This will ensure all our passwords and configuration options are synced and everything will work properly within the cluster.
  505. mail1 # cd ~
  506. mail1 # scp iRedMail-0.8.7/config mail2:~/iRedMail-0.8.7/
  508. You are now ready to begin the setup procedure on mail2. Use the same command from mail1 on mail2, and ignore any errors from MySQL about databases already existing (since they do!):
  510. mail2 # cd iRedMail-0.8.7/
  511. mail2 # MYSQL_SERVER='' MYSQL_ROOT_USER='admin_iredmail' MYSQL_GRANT_HOST='%' bash
  513. You will be informed that a config already exists; would you like to use it? Select "yes" to use the same settings as mail1 on mail2.
  515. A little bit of setup is required for ldirectord to manage the web page load balancing. Create a text file in the root of the web server (usually "/var/www") called "ldirectord.txt", containing the string "ldirectord"; as before, this was configured in the file on lbX:
  517. mailX # echo "ldirectord" > /var/www/ldirectord.txt
  519. As is good practice, drop back out of root to your unprivileged user now:
  521. mailX # exit
  522. mailX $
  525. 5b) Setting up LDAP multi-master replication
  527. Chapter source:
  529.         Once the install completes on mail2, we can proceed with configuring LDAP in a multi-master replication between mail1 and mail2 (and mail3 if you desire).
  531. Start by stopping the slapd service on both hosts:
  533. mailX $ sudo service slapd stop
  535. Edit the /etc/ldap/slapd.conf file on both hosts:
  537. mailX $ sudo vim /etc/ldap/slapd.conf
  539. Make the following changes:
  541.   i) under the "# Modules." section, add:
  543. moduleload  syncprov
  545.  ii) at the end of the file, add:
  547. # Multi master replication
  548. ServerID        1 "ldap://"
  549. ServerID        2 "ldap://"
  550. overlay         syncprov
  551. syncprov-checkpoint     10 1
  552. syncprov-sessionlog     100
  553. syncrepl        rid=1
  554.                 provider="ldap://mail1.local"
  555.                 type=refreshAndPersist
  556.                 interval=00:00:00:10
  557.                 retry="5 10 60 +"
  558.                 timeout=1
  559.                 schemachecking=off
  560.                 searchbase="dc=bonilan,dc=net"
  561.                 scope=sub
  562.                 bindmethod=simple
  563.                 binddn="cn=Manager,dc=example,dc=net"
  564.                 credentials="<LDAP rootdn password in plaintext>"
  565. syncrepl        rid=2
  566.                 provider="ldap://mail2.local"
  567.                 type=refreshAndPersist
  568.                 interval=00:00:00:10
  569.                 retry="5 10 60 +"  
  570.                 timeout=1
  571.                 schemachecking=off
  572.                 scope=sub
  573.                 searchbase="dc=bonilan,dc=net"
  574.                 bindmethod=simple
  575.                 binddn="cn=Manager,dc=example,dc=net"
  576.                 credentials="<LDAP rootdn password in plaintext>"
  577. MirrorMode      on
  579. [*] All servers are identical.
  580. [*] Using the "local" addresses in the "provider" lines allows the LDAP replication to occur over the local network for security. LDAP should be blocked at your firewall for the public addresses just like MySQL, unless required; each mailX host will look at its own local LDAP instance when accessing or modifying data.
  582. You can now start slapd on mail1:
  584. mail1 $ sudo service slapd start
  586. It should start normally; now, start it on mail2:
  588. mail2 $ sudo service slapd start
  590. Since you used the same "config" file for both, all the data should match up and you will now have a functioning, replicated LDAP setup. Test it out by using iRedAdmin to add data on mail1, and check if it exists on mail2. If it does, congratulations! You have a fully HA iRedMail setup.
  593. **
  594. 6. Final notes
  595. **
  597.         You now have a fully-functional cluster. All data is HA, and can tolerate the failure of any one set of nodes without interruption of service, either on the user or administrator side. You can now set up your first virtual domain ( with some users, and configure DNS for it:
  599.             IN      MX      1
  600.             IN      MX      1
  601.        IN      A
  602.        IN      A
  603.        IN      A
  604.         IN      A
  606. With this setup, your incoming mail will be redirected to one of either mail1 or mail2, where Postfix will filter and deliver it to the LDAP-backed mailbox of the domain user. Stored on NFS, that user can then access the mail using HTTP/S webmail, IMAPS, or POPS on the VIP, which will redirect to one of the two servers based on load and availability. The Dovecot session will use the syncronized MySQL backend to ensure consistency, and will read the data from the shared Maildir regardless of which real server the user is connected to. Try it out with a few users, and tinker with the settings to get it just perfect for you. And voilà! A HA mail solution in under 6000 words!
  609. **
  610. 7. Credits and Copyright
  611. **
  613.  * Joshua Boniface, the tinkerer and homelab geek who set this cluster up and documented the whole thing
  614.  * The iRedMail team for making this fantastic e-mail setup and management system
  615.  * The maintainers of a number of wonderful guides and manuals on how to configure the individual components
  616.  * YOU, for trying this out and leaving me your feedback
  618.     Copyright (C)  2014  JOSHUA BONIFACE.
  619.     Permission is granted to copy, distribute and/or modify this document
  620.     under the terms of the GNU Free Documentation License, Version 1.3
  621.     or any later version published by the Free Software Foundation;
  622.     with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
  623.     A copy of the license is included in the section entitled "GNU
  624.     Free Documentation License". ---
  626. Joshua Boniface is a Linux system administrator from Burlington, ON, Canada, specializing in Debian-based distributions. He can be found online under the handle "djbon2112", via his e-mail address joshua <at> boniface <dot> me, and at his website (under construction)
RAW Paste Data