SHARE
TWEET

An Ultra-HA, full Mult-Master E-mail cluster with iRedMail,

djbon2112 Sep 20th, 2014 (edited) 6,193 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. An Ultra-HA, full Mult-Master E-mail cluster with iRedMail, MariaDB, and IPVS
  2. by Joshua Boniface
  3.  
  4. -----
  5.  
  6. **
  7. 0. Edits
  8. **
  9.  
  10. 2015-05-11:
  11. As I made my cluster more expansive, I ran into an interesting problem that is caused by the simplicity of the setup in this guide, and was hidden from me because I was actually using a more complex setup. It concerns the ldirectord VIPs for accessing the database servers. I do apologize if you ran into any frustration due to this, as it drove me fairly mad when I ran into it myself. The changes are in the relevant section on ldirectord and minor changes to the cluster layout.
  12.  
  13. **
  14. 1. Introduction:
  15. **
  16.  
  17.         iRedMail is a very nifty piece of software. Setting up a full mail server on modern Linux is indeed possible; there are guides for every part of the system, preconfigured templates, and many-a mailing list post. However, iRedMail does something special: it makes it easy. Easy to install, easy to administer, and and easy to use. However, there are very few guides on how to deploy a *complete*, clustered iRedMail solution. Let's talk a bit about what I mean by that.
  18.  
  19.         I know e-mail, having deployed Debian-Linux-based carrier-grade mail platforms as part of my job. Setting up a cluster for production use, you want something that's fault-tolerant on every level. If you have 3 physical servers, you want to make sure you can keep your systems running even if you have to bring one down for maintenance, or due to a catastrophic hardware fault. You want to make sure you don't lose anything because one server crashed. However, many HA setups are not full multi-master; sure, you can lose a node, but it better not be server*1*. The setup I propose below addresses this: it doesn't matter which of your N nodes fails, you can always perform every task you need; and without proper monitoring, you might not even notice you have a problem! Some may think this is overkill, but the result is very compelling for anyone who values a 100% uptime!
  20.  
  21.         The setup I'm using is based on Debian Wheezy (7), the latest stable release of the Debian GNU/Linux operating system; 64-bit is of course recommended. I am very dilligent on separating services, for good reason: ease of manageability and flexibility, and as a result this guide uses slightly more VMs and IP addresses than one may expect. I will break down the cluster now so you have a better idea of how this is running.
  22.  
  23.         Please note that while this guide has parts that can be copy-pasted, and most specified shell commands and config files will work as intended, I do expect you to understand what you're doing, and RTFM if you don't; a decent knowledge of Linux System Administration is a must here. I don't go into any detail about creating your VMs, or any basic system administration tasks or commands, or what specific config options in files do. Also, all IP addresses/hostnames are fictitious and must be replaced, and anything in <> square brackets must be filled in with you own information. Finally, please note that I offer NO GUARANTEED SUPPORT FOR THIS GUIDE, though if you have a good question I'll probably answer it.
  24.  
  25.  
  26. **
  27. 2. The Cluster:
  28. **
  29.  
  30.         My home cluster is a fairly simple beast: there are two distinct hypervisors (hv1.example.net and hv2.example.net) running KVM, and a single file server (filer1.example.net). At this time, mostly due to budget reasons (it's a homelab, and those cost a lot of money in power!), I am not replicating to a second fileserver, and hence the backend Maildir storage is not HA in my setup. This can be acomplished in a huge number of ways (glusterFS, DRBD, manual sync) but is outside the scope of this guide. I assume that "filer1.example.net" is some device, providing a single NFS interface for backend storage of Maildirs.
  31.  
  32.         The Virtual Machines running on hv1.example.net and hv2.example.net are served via NFS from filer1, as are the Maildirs used for storing e-mail. This NFS is on a private network local to the servers, and this network also carries LDAP sync and Database traffic. The virtual machines are tied to a hypervisor: each *1* server is on hv1.example.net and each *2* server is on hv2.example.net. It's worth pointing out now that this cluster could easily be expanded to 3 hypervisors (and hence a *3* server for each service) if desired; this is recommended for the Database cluster in particular, however in my setup the filer1.example.net is the third, quorum-holding database server.
  33.  
  34.         I expect your setup to be slightly different. If so, just adapt this guide; I use consistent naming throughout (sed might be your friend here)!
  35.  
  36.  
  37. 2a) Virtual machines and networking
  38.  
  39.         The cluster comprises the following service VMs, all running Debian Wheezy:
  40.  
  41.           i) lbX, IPVS load balancers
  42.          ii) dbX, MariaDB/Galera MySQL servers
  43.         iii) mailX, iRedMail/LDAP servers
  44.  
  45.         Additionally, two VIP addresses for the load balancers is required:
  46.  
  47.          iv) lb0a, IPVS VIP for external mail services
  48.           v) lb0b, IPVS VIP for databases
  49.  
  50.         And one IP for the file server containing Maildirs:
  51.  
  52.          vi) filer1, NFS server
  53.  
  54.         The entire cluster therefore uses 9 IP addresses (ignoring the hypervisors, and any other VMs you might have set up). For the purposes of this guide, I assume two networks: "public", 1.1.1.0/24, and "private" , 10.1.1.0/24. You can omit either network and use a single private network; my proper "public" network uses routable public IPs, while the "private" network is unrouted, and certain replication traffic and NFS are kept on the "private" network for security. You can ignore this convention if you want, or even use Masquerade mode with IPVS, to hide all these services behind a single "public" IP and keep it all behind NAT. Whatever works for your environment! If you don't have a proper DNS setup, you can use this template in your "/etc/hosts" file on each host.
  55.  
  56.         # "Public"
  57.         1.1.1.11        filer1.example.net
  58.         1.1.1.21        lb0a.example.net # (VIP)
  59.         1.1.1.22        lb0a.example.net # (VIP)
  60.         1.1.1.13        lb1.example.net
  61.         1.1.1.14        lb2.example.net
  62.         1.1.1.15        db1.example.net
  63.         1.1.1.16        db2.example.net
  64.         1.1.1.17        mail1.example.net
  65.         1.1.1.18        mail2.example.net
  66.         # "Private"
  67.         10.1.1.11       filer1.local
  68.         10.1.1.13       lb1.local
  69.         10.1.1.14       lb2.local
  70.         10.1.1.15       db1.local
  71.         10.1.1.16       db2.local
  72.         10.1.1.17       mail1.local
  73.         10.1.1.18       mail2.local
  74.  
  75.         Note that there are no VIP in the "private" network: since all its services are load-balanced from the "public" IP, it is unnecessary for there to be a "local" VIP. I recommend firewalling the second VIP address to block MySQL traffic from the outside world if you are using a true public IP, since the MySQL cluster should not be visible to the outside world.
  76.  
  77.  
  78. 2b) A note on example conventions
  79.  
  80. Thoughout this guide, when command-lines are given, the following rules will be held:
  81.  
  82.   i) the beginning of the prompt will indicate the server name, either as:
  83.         server1 # <command>
  84.      for a specific server ID, or:
  85.         serverX # <command>
  86.      for all servers in that category, or even:
  87.         serva1, servb2 # <command>
  88.      for two specific server names
  89.  ii) the seperator character shall be isolated on both sides by a space (for ease of copying a single command, but
  90.      discouraging block copying) and will consist of:
  91.         # - for a root-level account
  92.         $ - for an unprivileged account
  93. iii) for simplicity, most commands in this guide are written as an unprivileged user with "sudo" prepended;
  94.      commands that require the actual *root* account (e.g. the iRedMail.sh setup script) will use # instead
  95.  iv) when editing a text file, the raw contents from server1 will be presented after the command (usually 'sudo
  96.      vim'), followed by a 'diff' of the differences between server1 and server2, if necessary; one can extrapolate
  97.      the third server or any other differences if desired
  98.   v) any comments regarding a text file will follow the output and diff, prepended by a [*] for each comment
  99.  
  100.  
  101. **
  102. 3. Setting up IPVS, ldirectord, and keepalived
  103. **
  104.  
  105. Chapter source: http://www.ultramonkey.org/papers/lvs_tutorial/html/
  106.  
  107.         The first and probably easiest part of this cluster is the load balancing configuration. It is a very straightforward setup, with 2 load-balancers sharing 2 VIP addresses: if contact is lost, keepalived moves the VIPs between the two servers on a weighted basis (lb1 is prefered to lb2).
  108.  
  109. Start by installing the required packages on both hosts.
  110.  
  111. lbX $ sudo apt-get update
  112. lbX $ sudo apt-get upgrade
  113. lbX $ sudo apt-get install ipvsadm ldirectord keepalived
  114.  
  115.  
  116. 3a) keepalived
  117.  
  118.         Begin by editing the keepalivd configuration. This will set up the VIPs between the two load balancers, and allow them to fail over from lb1 to lb2 in the event lb1 goes down, thus preserving services to anyone connecting to the cluster through this IP.
  119.  
  120. lbX $ sudo vim /etc/keepalived/keepalived.conf
  121.  
  122. vrrp_instance VI_1 {
  123.         state MASTER
  124.         interface eth0
  125.         virtual_router_id 1
  126.         priority 200
  127.         authentication {
  128.                 auth_type PASS
  129.                 auth_pass mySuperSecretPassw0rd
  130.         }
  131.         virtual_ipaddress {
  132.                 1.1.1.21/24;
  133.                 1.1.1.22/24;
  134.         }
  135. }
  136.  
  137. 5c5
  138. <         priority 200
  139. ---
  140. >         priority 100
  141.  
  142. [*] The adjusted priority on lb2 allows lb1 to take precidence and prevent flapping between the two load balancers. If you have a third lbX host, you can make its priority something less than 100 to ensure it will be last in the chain.
  143.  
  144. Restart the keepalived service on both hosts:
  145.  
  146. lbX $ sudo service keepalived restart
  147.  
  148. You should now see the VIPs in the list of IP addresses on lb1:
  149.  
  150. lb1 $ ip a
  151. [...]
  152. eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
  153.     link/ether 11:11:11:11:11:11 brd ff:ff:ff:ff:ff:ff
  154.     inet 1.1.1.13/24 brd 1.1.1.255 scope global eth0
  155.     inet 1.1.1.21/24 scope global secondary eth0
  156.     inet 1.1.1.22/24 scope global secondary eth0
  157. [...]
  158.  
  159. Now stop keepalived on lb1, and observe lb2: the IP addresses will transfer after a second or two:
  160.  
  161. lb1 $ sudo service keepalived stop
  162.  
  163. lb2 $ ip a
  164. [...]
  165. eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
  166.     link/ether 11:11:11:22:22:22 brd ff:ff:ff:ff:ff:ff
  167.     inet 1.1.1.14/24 brd 1.1.1.255 scope global eth0
  168.     inet 1.1.1.21/24 scope global secondary eth0
  169.     inet 1.1.1.22/24 scope global secondary eth0
  170. [...]
  171.  
  172. You can then restart the keepalived service on lb1 and the IP addresses will return to it, exactly as expected.
  173.  
  174.  
  175. 3b) ldirectord
  176.  
  177.         Next is the ldirectord configuration. ldirectord is a load balancer using IPVS, the Linux kernel virtual server, to distribute traffing entering on a VIP between a number of "real servers", . It contains a number of different load-balancing and routing options, however for our purposes, with a "public" network, we will use the 'routed' mode, whereby traffic is directly routed from the VIP to the real server, which has configured on a loopback interface the VIP, allowing traffic to be sent directly back to the client from the real server, reducing load on the load balancers. In effect, the routers simply keep track of incoming packets while the outgoing packets flow right to the client.
  178.  
  179. ldirectord works by performing regular health checks on the real servers; if one is found to be non-working, it is removed from the IPVS configuration, thus preventing clients from being directed to a dead server. Once the service has been restored, ldirectord re-adds the real server to the IPVS configuration, and load-balancing resumes.
  180.  
  181. The following ldirectord.cf file contains all the services that will be provided in HA mode for client access, the list of which is: MySQL, HTTP/S, IMAPS, POPS, and SMTPSUB. I don't allow unsecured access via IMAP or POP3 directly to my mail servers, but you can add these services if desired.
  182.  
  183. lbX $ sudo vim /etc/ldirectord.cf
  184.  
  185. logfile="daemon"
  186. fallbackcommand=""
  187. failurecount=3
  188. checkinterval=5
  189. fork=yes
  190.  
  191. # MySQL database to db1/db2
  192. virtual=1.1.1.22:3306
  193.         real=1.1.1.15:3306 gate
  194.         real=1.1.1.16:3306 gate
  195.         service=mysql
  196.         scheduler=sh
  197.         login="monitor"
  198.         passwd="monitoringPassw0rd"
  199.         request="SELECT * from monitoring.monitoring;"
  200. # Mail services to mail1/mail2
  201. virtual=1.1.1.21:80
  202.         real=1.1.1.17:80 gate
  203.         real=1.1.1.18:80 gate
  204.         service=http
  205.         scheduler=sh
  206.         request="ldirectord.txt"
  207.         receive="ldirectord"
  208. virtual=1.1.1.21:443
  209.         real=1.1.1.17:443 gate
  210.         real=1.1.1.18:443 gate
  211.         service=https
  212.         scheduler=sh
  213.         request="ldirectord.txt"
  214.         receive="ldirectord"
  215. virtual=1.1.1.21:993
  216.         real=1.1.1.17:993 gate
  217.         real=1.1.1.18:993 gate
  218.         service=imaps
  219. virtual=1.1.1.21:995
  220.         real=1.1.1.17:995 gate
  221.         real=1.1.1.18:995 gate
  222.         service=pops
  223. virtual=1.1.1.21:465
  224.         real=1.1.1.17:465 gate
  225.         real=1.1.1.18:465 gate
  226.         service=smtp
  227.  
  228. [*] Both servers are identical.
  229. [*] Service checks for MySQL and HTTP/S will be addressed in their relevant sections.
  230.  
  231. Reload the ldirectord service, and use "ipvsadm" to view the resulting IPVS configuration (IP-to-hostname translation is used, if you don't have reverse DNS configured you will see IP addresses):
  232.  
  233. lbX $ sudo service ldirectord restart
  234. lbX $ sudo ipvsadm
  235. IP Virtual Server version 1.2.1 (size=4096)
  236. Prot LocalAddress:Port Scheduler Flags
  237.   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  238. TCP  lb0b.example.net:mysql sh
  239.   -> db1.example.net:mysql        Route   1      0          0        
  240.   -> db2.example.net:mysql        Route   1      0          0
  241. TCP  lb0a.example.net:http sh
  242.   -> mail1.example.net:http       Route   1      0          0        
  243.   -> mail2.example.net:http       Route   1      0          0
  244. TCP  lb0a.example.net:https sh
  245.   -> mail1.example.net:https      Route   1      0          0        
  246.   -> mail2.example.net:https      Route   1      0          0
  247. TCP  lb0a.example.net:imaps wrr
  248.   -> mail1.example.net:imaps      Route   1      0          0        
  249.   -> mail2.example.net:imaps      Route   1      0          0
  250. TCP  lb0a.example.net:pops wrr
  251.   -> mail1.example.net:pops       Route   1      0          0        
  252.   -> mail2.example.net:pops       Route   1      0          0
  253. TCP  lb0a.example.net:submission wrr
  254.   -> mail1.example.net:submission Route   1      0          0        
  255.   -> mail2.example.net:submission Route   1      0          0
  256.  
  257. However, since you have not yet configured any services, there will be no real servers in your output, only the lines containing "lb0x.example.net":
  258.  
  259. IP Virtual Server version 1.2.1 (size=4096)
  260. Prot LocalAddress:Port Scheduler Flags
  261.   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  262. TCP  lb0b.example.net:mysql sh
  263. TCP  lb0a.example.net:http sh
  264. TCP  lb0a.example.net:https sh
  265. TCP  lb0a.example.net:imaps wrr
  266. TCP  lb0a.example.net:pops wrr
  267. TCP  lb0a.example.net:submission wrr
  268.  
  269. Once this guide is done, compare the resulting output from "ipvsadm" to the above output, and you should see it match!
  270.  
  271. This concludes the configuration required on the load balancers themselves. However, one more piece of configuration must be done to *each* real server: it must have the VIP address for the host added to a loopback interface to allow services on the server to use that address to reply to clients. This is a required part of the "direct-routing" mode used in IPVS. If you are using an alternate routing mode (for example Masquerade), you do not need this step. On each dbX and mailX host:
  272.  
  273. serverX $ sudo vim /etc/network/interfaces
  274.  
  275. # Loopback interface
  276. auto lo
  277. iface lo inet loopback
  278.  
  279. # IPVS-DR loopback interface
  280. auto lo:0
  281. iface lo:0 inet static
  282.         address 1.1.1.21
  283.         netmask 255.255.255.255
  284.         pre-up sysctl -w net.ipv4.conf.all.arp_ignore=1
  285.         pre-up sysctl -w net.ipv4.conf.all.arp_announce=2
  286.  
  287. # Other interfaces, server-specific...
  288. [...]
  289.  
  290. [*] All servers are identical, except dbX which use 1.1.1.22/24 as the IP.
  291.  
  292. This concludes the configuration of the load balancer setup, and the VIP that will direct requests to the client machines.
  293.  
  294. ---
  295. EDIT 2015-05-11: WHY DO I NEED TWO VIPs?
  296.  
  297. The simple reason is: because of how networking works. In the original version of this guide, I recommended setting this up with a single VIP: that VIP would be present on four servers: db1, db2, mail1, and mail2. The idea was that outgoing database requests from mailX would hit the VIP on lb0, then be redirected to one of the dbX hosts, like so:
  298.  
  299. mail1 -> lb0 -> db?
  300.  
  301. However, by having the *same* VIP address on the mailX hosts that we do on the dbX hosts, we run into a major problem: when mail1 tries to talk to lb0, what actually happens is it tries to talk to itself, on it's lo:0 interface! And naturally, since MySQL isn't running on mail1 (or mail2), it will fail to connect!
  302.  
  303. There are a number of ways to solve this problem, but truthfully to preserver the existing load balancing setup, the simplest is to just add a second VIP dedicated to the database cluster. You can then alias "db.example.net" to "lb0b" and continue with the rest of the guide unmodified.
  304. ---
  305.  
  306. **
  307. 4. Setting up MariaDB and Galera multi-master SQL
  308. **
  309.  
  310. Chapter source: https://blog.mariadb.org/installing-mariadb-galera-cluster-on-debian-ubuntu/
  311.  
  312.         As seen above, one of the load-balanced services is MySQL. Databases are used extensively in e-mail servers: they hold information about active accounts, sessions, filter policies; the list goes on. The services of the dbX servers could be integrated into the mailX servers themselves, however in my usage it makes more sense to separate them. You can easily run all of the following on the mailX servers and reduce your IP usage by two if you so desire (just don't forget to edit the ldirectord.cf file in Chapter 3 to match!)
  313.  
  314. The MySQL cluster will be provided by MariaDB, a community-driven fork of Oracle's MySQL, and headed by the original developers of MySQL. It is combined with the Galera replication engine to allow a multi-master cluster than can be load-balanced by IPVS. I am using version 5.5 for maximum compatibility, though the newer releases could be used as well. To prevent split-brain, we also use a third host in the Galera cluster, which will be provided by the filer1 server; if you are using this guide to set up a 3-server cluster, you can exclude that host as quorum will be provided by 3 dbX servers. Run all commands below on filer1 as well as dbX.
  315.  
  316. Start by adding the MariaDB sources (other mirrors can be found at https://downloads.mariadb.org/mariadb/repositories/) into your apt configuration:
  317.  
  318. dbX $ sudo apt-get install python-software-properties
  319. dbX $ sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xcbcb082a1bb943db
  320. dbX $ sudo add-apt-repository 'deb http://mariadb.mirror.rafal.ca/repo/5.5/debian wheezy main'
  321.  
  322. Then update and install the required packages:
  323.  
  324. dbX $ sudo apt-get update
  325. dbX $ sudo apt-get install rsync galera mariadb-galera-server
  326.  
  327. You will be asked for a root password during setup; ensure it is identical on all hosts. Once installed, stop the mysql process as we need it off for the next steps.
  328.  
  329. dbX $ sudo service mysql stop
  330.  
  331. The Galera configuration below is extremely simple; read the Galera documentation for more advanced configuration options. Set the local IP addresses of the cluster members in the "wsrep_cluster_address" line, to keep replication traffic on the unrouted local network. You can also set the "wsrep_cluster_name" to a new value; this is in effect a shared secret for the cluster.
  332.  
  333. dbX $ sudo vim /etc/mysql/conf.d/galera.cnf
  334.        
  335. [mysqld]
  336. # MySQL settings
  337. binlog_format=ROW
  338. default-storage-engine=innodb
  339. innodb_autoinc_lock_mode=2
  340. query_cache_size=0
  341. query_cache_type=0
  342. bind-address=0.0.0.0
  343. # Galera settings
  344. wsrep_provider=/usr/lib/galera/libgalera_smm.so
  345. wsrep_cluster_name="db_cluster"
  346. wsrep_cluster_address="gcomm://10.1.1.11,10.1.1.15,10.1.1.16"
  347. wsrep_sst_method=rsync
  348.  
  349. [*] All servers are identical.
  350.  
  351. Also clone the /etc/mysql/debian.cnf file from db1 to the other database hosts; this will, combined with the tweaks below, prevent "debian-sys-maint" access denied warnings when starting the other cluster nodes.
  352.  
  353. Warning: Before we continue, I have discovered a bug in this setup. Because of the IPVS-DR loopback, the Galera cluster will sometimes fail to start on the second or third node of the cluster. The reasons I do not completely understand. To mitigate this however, I made a modification to the "/etc/init.d/mysql" initscript to add an "ifdown lo:0" and corresponding "ifup lo:0" at the beginning and end, respectively, of the "start" function. I recommend doing this to save you hours of headaches!
  354.  
  355. Once the configuration is in place on all nodes, we can start the cluster on the first node:
  356.  
  357. db1 $ sudo service mysql start --wsrep-new-cluster
  358.  
  359. The "--wsrep-new-cluster" directive creates a new active cluster; if all hosts in the Galera cluster go down, you will need to execute this command on a node again to start up the cluster. Data is of course preserved when running this command, and the host it is run on will become the "primary" sync source for the other members of the cluster.
  360.  
  361. On the remaining nodes, start the MySQL service normally:
  362.  
  363. db2,filer1 $ sudo service mysql start
  364.  
  365. If all goes well, they will connect to the cluster master, and data will synchronize. Check the number of nodes in the cluster with:
  366.  
  367. db1 $ mysql -u root -p -e 'SELECT VARIABLE_VALUE as "cluster size" FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME="wsrep_cluster_size"'
  368. +--------------+
  369. | cluster size |
  370. +--------------+
  371. | 3            |
  372. +--------------+
  373.  
  374. There are a number of configuration tweaks that must be performed to properly use the MySQL cluster as expected. Enter the database server, and:
  375.  
  376. db1 $ mysql -u root -p
  377. Enter password:
  378. Welcome to the MariaDB monitor.  Commands end with ; or \g.
  379. Your MariaDB connection id is 104893
  380. Server version: 5.5.38-MariaDB-1~wheezy-wsrep-log mariadb.org binary distribution, wsrep_25.10.r3997
  381. Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
  382. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
  383. MariaDB [(none)]>
  384.  
  385.   i) Create a "new" root user with global host access. Yes, you can restrict this per-host, but that gets messy fast
  386.      and if you have a *properly secured* network, you shouldn't have to worry about this too much.
  387.  
  388. MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION IDENTIFIED BY '<MySQL root password>';
  389. MariaDB [(none)]> GRANT PROXY ON ''@'' TO 'root'@'%' WITH GRANT OPTION;
  390.  
  391.      Test that this user works by logging in to another MySQL shell, and if it works fine, drop all "old" root users:
  392.  
  393. MariaDB [(none)]> SELECT User,Host from mysql.user;
  394. [view the resulting list of users]
  395. MariaDB [(none)]> DROP USER 'root'@'db1';
  396. MariaDB [(none)]> DROP USER 'root'@'db2';
  397. [etc.]
  398.  
  399.  ii) Create a "new" debian-sys-maint user, with slighly more restricted access than the root user; again this user
  400.      should be for the global host for simplicity.
  401.  
  402. MariaDB [(none)]> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER ON *.* TO 'debian-sys-maint'@'%' IDENTIFIED BY '<debian-sys-maint password from /etc/mysql/debian.cnf>' WITH GRANT OPTION;
  403.  
  404.      Make sure that the "/etc/mysql.debian.cnf" file is identical between all the dbX nodes. And like the root user, drop all "old" debian-sys-maint users:
  405.  
  406. MariaDB [(none)]> DROP USER 'debian-sys-maint'@'db1';
  407. MariaDB [(none)]> DROP USER 'debian-sys-maint'@'db2';
  408. [etc.]
  409.  
  410. Attempt to stop the mysql service on any one node; it should suceed without any warnings or errors about permission denied; restart it and resume configuration.
  411.  
  412. Now we will add some data to the cluster and observe its replication. The data used is, conveniently, the monitoring framework required by ldirectord.
  413.  
  414. db1 $ mysql -u root -p
  415. MariaDB [(none)]>
  416.  
  417. Begin by creating a new database called 'monitoring'; these values were set in Chapter 3, in ldirectord.cf:
  418.  
  419. MariaDB [(none)]> CREATE DATABASE monitoring;
  420.  
  421. Create a new user, 'monitor', identified by the password 'monitoringPassw0rd', and grant select access to the 'monitoring' database:
  422.  
  423. MariaDB [(none)]> GRANT SELECT ON monitoring.* TO 'monitor'@'%' IDENTIFIED BY 'monitoringPassw0rd';
  424.  
  425. Now, change into the monitoring database, and create a table called "monitoring" containing some data:
  426.  
  427. MariaDB [(none)]> USE monitoring;        
  428. MariaDB [monitoring]> CREATE TABLE monitoring (data VARCHAR(1));
  429. MariaDB [monitoring]> INSERT INTO monitoring (data) VALUES ("X");
  430. MariaDB [monitoring]> SELECT * FROM monitoring.monitoring;
  431. +------+
  432. | data |
  433. +------+
  434. | X    |
  435. +------+
  436. MariaDB [monitoring]> quit
  437. Bye
  438.  
  439. You have now set up the monitoring table that the ldirectord daemon will connect to and attempt to judge your hosts' health. If everything is configured and working right, you should now see the real servers in the output of "ipvsadm" on lbX:
  440.  
  441. lbX $ sudo ipvsadm
  442. IP Virtual Server version 1.2.1 (size=4096)
  443. Prot LocalAddress:Port Scheduler Flags
  444.   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
  445. TCP  lb0.example.net:mysql wrr
  446.   -> db1.example.net:mysql        Route   1      0          0        
  447.   -> db2.example.net:mysql        Route   1      0          0
  448. [...]
  449.  
  450. You can see the data replication by checking any other node: you should be able to see the monitoring database and its table. But if both your dbX servers are in the IPVS configuration above, you should be good to go! Now you can access the databases, load-balanced between the two dbX VMs, from the VIP address at lb0.example.net. I recommend aliasing/pointing "db.example.net" to "lb0.example.net" now as this will be used as a referenced name later.
  451.  
  452. A note about filer1: When setting up the DB, we used filer1 as our third, quorum-holding host. However, as that is not a proper "database" server, it is NOT added to the load-balanced cluster. It is, in effect, a write-only copy of the data to preserve quorum. If you are using three physical servers, and hence three dbX servers, you should be able to use just those 3 to maintain quorum, and load-balance over all 3, in which case you can avoid putting any MySQL on the filer1 server.
  453.  
  454. This concludes the configuration of the database cluster, which is now ready for data to be added from the mailX servers, via the load-balanced VIP address at lb0.example.net.
  455.  
  456.  
  457. **
  458. 5. Highly-available, LDAP-backed iRedMail e-mail
  459. **
  460.  
  461. Chapter source: iRedAdmin Wiki/Existing Tutorials
  462.  
  463.         And now for the Pièce de résistance, the whole reason for this tutorial: the HA iRedMail cluster! For this setup, we will be using a multi-master LDAP backend to store user data; each mail server will connect to its own, local, LDAP database and those databases will be synchronized between the two servers. This allows iRedAdmin management, as well as user access, to be passed through the load balancer: if a mail server goes down, regardless of which one, the mail administrator(s) can still make changes and continue working as if nothing happened to the infrastructure backend; let the sysadmins worry about that!
  464.  
  465. Note that mirrormode LDAP multi-master clusters are ideally used with a "preferred" master. For this reason, I don't recommend *actually* using the VIP address to access iRedAdmin under normal situations: manage from mail1, and if needed, manage from mail2 if mail1 is down. This helps preserve consistency so that you can trust one particular host if an LDAP split-brain happens.
  466.  
  467.  
  468. 5a) iRedMail installation with the HA MySQL backend
  469.  
  470. To begin, become root on both mail servers:
  471.  
  472. mailX $ sudo su
  473. mailX #
  474.  
  475. Now download and extract the latest version of iRedMail, at the time of writing iRedMail-0.8.7:
  476.  
  477. mailX # wget iRedMail-0.8.7.tar.bz2
  478. mailX # tar -xvjf iRedMail-0.8.7.tar.bz2
  479.  
  480. Before we begin installing iRedMail, add a fully-privileged user to the database server that can be used by the install script to set up the databases. This user can be removed after installation; alternatively, you could use root but this is not recommended.
  481.  
  482. db1 # mysql -u root -p
  483. MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'admin_iredmail'@'%' WITH GRANT OPTION IDENTIFIED BY '<password>';
  484.  
  485. Next, mount the NFS Maildir storage from filer1 on both mailX hosts (I prefer the default "/var/vmail"); ensure you add it to "/etc/fstab" as well:
  486.  
  487. mailX # mkdir /var/vmail
  488. mailX # chattr +i /var/vmail
  489. mailX # mount -t nfs -o vers=3,noatime,nodiratime,noexec filer1.local:/srv/var/vmail /var/vmail
  490. mailX # echo "filer1.local:/srv/var/vmail /var/vmail nfs vers=3,noatime,nodiratime,noexec 0 0" >> /etc/fstab
  491.  
  492. Configuration should begin on mail1 first: this will allow us to generate an iRedMail installer 'config' file, which we will then use to ensure mail2 is configured with the same settings.
  493.  
  494. Start the iRedMail installer with variables specifying the database host, user, and grant host (in our case, '%' for simplicity in our MySQL users):
  495.  
  496. mail1 # cd iRedMail-0.8.7/
  497. mail1 # MYSQL_SERVER='db.example.net' MYSQL_ROOT_USER='admin_iredmail' MYSQL_GRANT_HOST='%' bash iRedMail.sh
  498.  
  499. Follow the directions as per the standard iRedMail setup procedure. In particular, choose an LDAP backend, and choose the NFS directory for the Maildir storage. Also ensure that you save any password you entered: these will eventually be the cluster master passwords. During setup, you will be asked for the password for "admin_iredmail" you set above in order for the installer to access the MySQL cluster. Also, don't use an additional domain when asked for your first virtual domain: use "example.net". This will simplify our deployment and allow you to add actual domains to the full cluster later.
  500.  
  501. Once the iRedMail setup completes, your first node will be operational! Feel free to test it out, and inspect the database servers to confirm that the data for the iRedMail server was properly added to the MySQL cluster backend and is replicating between the hosts as expected.
  502.  
  503. Next, copy the "config" file from the iRedMail installer directory over to the second server. This will ensure all our passwords and configuration options are synced and everything will work properly within the cluster.
  504.  
  505. mail1 # cd ~
  506. mail1 # scp iRedMail-0.8.7/config mail2:~/iRedMail-0.8.7/
  507.  
  508. You are now ready to begin the setup procedure on mail2. Use the same command from mail1 on mail2, and ignore any errors from MySQL about databases already existing (since they do!):
  509.  
  510. mail2 # cd iRedMail-0.8.7/
  511. mail2 # MYSQL_SERVER='db.example.net' MYSQL_ROOT_USER='admin_iredmail' MYSQL_GRANT_HOST='%' bash iRedMail.sh
  512.  
  513. You will be informed that a config already exists; would you like to use it? Select "yes" to use the same settings as mail1 on mail2.
  514.  
  515. A little bit of setup is required for ldirectord to manage the web page load balancing. Create a text file in the root of the web server (usually "/var/www") called "ldirectord.txt", containing the string "ldirectord"; as before, this was configured in the ldirectord.cf file on lbX:
  516.  
  517. mailX # echo "ldirectord" > /var/www/ldirectord.txt
  518.  
  519. As is good practice, drop back out of root to your unprivileged user now:
  520.  
  521. mailX # exit
  522. mailX $
  523.  
  524.  
  525. 5b) Setting up LDAP multi-master replication
  526.  
  527. Chapter source: http://www.openldap.org/doc/admin24/replication.html
  528.  
  529.         Once the install completes on mail2, we can proceed with configuring LDAP in a multi-master replication between mail1 and mail2 (and mail3 if you desire).
  530.  
  531. Start by stopping the slapd service on both hosts:
  532.  
  533. mailX $ sudo service slapd stop
  534.  
  535. Edit the /etc/ldap/slapd.conf file on both hosts:
  536.  
  537. mailX $ sudo vim /etc/ldap/slapd.conf
  538.  
  539. Make the following changes:
  540.  
  541.   i) under the "# Modules." section, add:
  542.  
  543. moduleload  syncprov
  544.  
  545.  ii) at the end of the file, add:
  546.  
  547. # Multi master replication
  548. ServerID        1 "ldap://mail1.example.net"
  549. ServerID        2 "ldap://mail2.example.net"
  550. overlay         syncprov
  551. syncprov-checkpoint     10 1
  552. syncprov-sessionlog     100
  553. syncrepl        rid=1
  554.                 provider="ldap://mail1.local"
  555.                 type=refreshAndPersist
  556.                 interval=00:00:00:10
  557.                 retry="5 10 60 +"
  558.                 timeout=1
  559.                 schemachecking=off
  560.                 searchbase="dc=bonilan,dc=net"
  561.                 scope=sub
  562.                 bindmethod=simple
  563.                 binddn="cn=Manager,dc=example,dc=net"
  564.                 credentials="<LDAP rootdn password in plaintext>"
  565. syncrepl        rid=2
  566.                 provider="ldap://mail2.local"
  567.                 type=refreshAndPersist
  568.                 interval=00:00:00:10
  569.                 retry="5 10 60 +"  
  570.                 timeout=1
  571.                 schemachecking=off
  572.                 scope=sub
  573.                 searchbase="dc=bonilan,dc=net"
  574.                 bindmethod=simple
  575.                 binddn="cn=Manager,dc=example,dc=net"
  576.                 credentials="<LDAP rootdn password in plaintext>"
  577. MirrorMode      on
  578.  
  579. [*] All servers are identical.
  580. [*] Using the "local" addresses in the "provider" lines allows the LDAP replication to occur over the local network for security. LDAP should be blocked at your firewall for the public addresses just like MySQL, unless required; each mailX host will look at its own local LDAP instance when accessing or modifying data.
  581.  
  582. You can now start slapd on mail1:
  583.  
  584. mail1 $ sudo service slapd start
  585.  
  586. It should start normally; now, start it on mail2:
  587.  
  588. mail2 $ sudo service slapd start
  589.  
  590. Since you used the same "config" file for both, all the data should match up and you will now have a functioning, replicated LDAP setup. Test it out by using iRedAdmin to add data on mail1, and check if it exists on mail2. If it does, congratulations! You have a fully HA iRedMail setup.
  591.  
  592.  
  593. **
  594. 6. Final notes
  595. **
  596.  
  597.         You now have a fully-functional cluster. All data is HA, and can tolerate the failure of any one set of nodes without interruption of service, either on the user or administrator side. You can now set up your first virtual domain (example.net) with some users, and configure DNS for it:
  598.  
  599. example.net             IN      MX      1 mail1.example.net
  600. example.net             IN      MX      1 mail2.example.net
  601. mail.example.net        IN      A       1.1.1.12
  602. smtp.example.net        IN      A       1.1.1.12
  603. imap.example.net        IN      A       1.1.1.12
  604. pop.example.net         IN      A       1.1.1.12
  605.  
  606. With this setup, your incoming mail will be redirected to one of either mail1 or mail2, where Postfix will filter and deliver it to the LDAP-backed mailbox of the domain user. Stored on NFS, that user can then access the mail using HTTP/S webmail, IMAPS, or POPS on the VIP, which will redirect to one of the two servers based on load and availability. The Dovecot session will use the syncronized MySQL backend to ensure consistency, and will read the data from the shared Maildir regardless of which real server the user is connected to. Try it out with a few users, and tinker with the settings to get it just perfect for you. And voilà! A HA mail solution in under 6000 words!
  607.  
  608.  
  609. **
  610. 7. Credits and Copyright
  611. **
  612.  
  613.  * Joshua Boniface, the tinkerer and homelab geek who set this cluster up and documented the whole thing
  614.  * The iRedMail team for making this fantastic e-mail setup and management system
  615.  * The maintainers of a number of wonderful guides and manuals on how to configure the individual components
  616.  * YOU, for trying this out and leaving me your feedback
  617.  
  618.     Copyright (C)  2014  JOSHUA BONIFACE.
  619.     Permission is granted to copy, distribute and/or modify this document
  620.     under the terms of the GNU Free Documentation License, Version 1.3
  621.     or any later version published by the Free Software Foundation;
  622.     with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
  623.     A copy of the license is included in the section entitled "GNU
  624.     Free Documentation License". --- https://gnu.org/licenses/fdl.html
  625.        
  626. Joshua Boniface is a Linux system administrator from Burlington, ON, Canada, specializing in Debian-based distributions. He can be found online under the handle "djbon2112", via his e-mail address joshua <at> boniface <dot> me, and at his website (under construction) http://www.boniface.me.
RAW Paste Data
Pastebin PRO BLACK FRIDAY Special!
Get 60% OFF on Pastebin PRO accounts!
Top