Support: 1-800-961-4454
Sales Chat
1-800-961-2888

High Availability (HA) With Galera For Rackspace Private Cloud

EDITOR’S NOTE: This post is purely for educational purposes only. Rackspace Private Cloud does not officially support Galera.

We’ve been hearing from our customers a lot lately about High Availability (HA). In a recent blog post, I discussed how implementing HA for MySQL in Rackspace Private Cloud is simple and works well with two controller nodes. HA will also be the focus of my talk at OpenStack Summit Hong Kong next month.

In this post, I will leverage the HA as implemented in Rackspace Private Cloud and outline steps to replace the default Active/Passive MySQL master-master replication with Active/Active Galera multi-master replication. As a result of HA being Active/Active, it’s possible to scale out MySQL with methods similar to scaling out the API calls, such as Keystone, Nova and so on.

NOTE: Rackspace Private Cloud does not officially support Galera. Even though I have ran through some of the basic functionality, I have not done rigorous testing by any means.

What is Galera?

Galera is a multi-master replication technique for MySQL and works in conjunction with the wsrep (Write Set Replication) APIs. It effectively treats a cluster of MySQL nodes as one master so that reads and writes are possible on any node.  Using this method it is possible to achieve both HA and scale out for MySQL.

The tutorial at Severalnines goes into greater detail.

Galera for Rackspace Private Cloud

As I noted previously, Rackspace Private Cloud installs HAProxy, Keepalived and MySQL in addition to other OpenStack software using Rackspace’s Chef cookbooks. I will start with a HA install of Rackspace Private Cloud with two controller nodes and one or more compute node(s). After verifying this installation and running a few commands, we will replace the default Rackspace Private Cloud MySQL with a Galera install. We will continue to leverage HAProxy and Keepalived (which also provides the VIP).

This is a manual process, which is not officially supported. The goal is to illustrate that it’s possible to replace the default MySQL with Galera. There are limitations to this approach; for example, you will not be able to add additional compute nodes using the Rackspace cookbooks, since they expect the default MySQL install. Also, running some of the Chef commands will possibly overwrite the manual configuration changes.

Here are steps for installing Galera and to use that as the database for a Rackspace Private Cloud installation of OpenStack.

  • Step 1: Install Rackspace Private Cloud on two controllers with HA mode. HAproxy, Keepalived and VRRP are installed as part of the process.
  • Step 2: Install Galera (with wsrep) on three separate nodes.
  • Step 3: Use mysqldump to transfer MySQL data from controller nodes to Galera node.
  • Step 4: Grant privileges to OpenStack (nova, glance, etc.) and other users.
  • Step 5: Update Keepalived and HAproxy configuration files on controller nodes.
  • Step 6: Stop MySQL services on controller nodes and restart.

Step 1: Install Rackspace Private Cloud

I used Vagrant and VirtualBox and followed the instructions in this blog substituting the appropriate static IP addresses shown in the table below.

I used the roles ha-controller1, ha-controller2 and single-compute for l1, l2 and c1 respectively.

I verified the install by running a few commands. We will run the same commands after we install Galera. The following four commands will each produce a list of services, showing that the installation is working.

nova-manage service list
keystone service-list
glance image-list
nova service-list

The following list is an example of the output you can expect:

+------------------+------+----------+---------+-------+----------------------------+
| Binary           | Host | Zone     | Status  | State | Updated_at                 |
+------------------+------+----------+---------+-------+----------------------------+
| nova-cert        | l1   | internal | enabled | up    | 2013-10-17T18:39:23.000000 |
| nova-cert        | l2   | internal | enabled | up    | 2013-10-17T18:39:27.000000 |
| nova-compute     | c1   | nova     | enabled | up    | 2013-10-17T18:39:29.000000 |
| nova-conductor   | l1   | internal | enabled | up    | 2013-10-17T18:39:24.000000 |
| nova-conductor   | l2   | internal | enabled | up    | 2013-10-17T18:39:27.000000 |
| nova-consoleauth | l1   | internal | enabled | up    | 2013-10-17T18:39:24.000000 |
| nova-consoleauth | l2   | internal | enabled | up    | 2013-10-17T18:39:27.000000 |
| nova-network     | c1   | internal | enabled | up    | 2013-10-17T18:39:26.000000 |
| nova-scheduler   | l1   | internal | enabled | up    | 2013-10-17T18:39:24.000000 |
| nova-scheduler   | l2   | internal | enabled | up    | 2013-10-17T18:39:27.000000 |
+------------------+------+----------+---------+-------+----------------------------+

Step 2: Install Galera

I created the following nodes for the Galera install. The ClusterControl server (referred to as cmon) will be used to run the installation commands:

I used the Galera Configurator by Severalnines for the process and provided the following values, accepting the defaults for the rest:

  • Vendor: Codership.
  • On the “General Settings” tab:
    • Infrastructure: on-premise
    • Operating System: Ubuntu 12.04
    • OS User: root
  • On the “Configure Storage” tab:
    • System Memory: 512MB
    • Skip DNS Resolve: Yes
  • On the “Configure Directories” tab:
    • Cluster Control Server: 192.168.236.50
    • Server-id: 192.168.236.51,  192.168.236.52 and 192.168.236.53 for server-id 1,2 and 3 respectively.

Finally, I entered my email address on the “Review Config” tab and clicked the “Generate Deployment Scripts” button.

After the configuration was complete, I was able to log onto the cmon server and proceed.

On the server cmon, you will unpack the file provided and issue the following commands. You will receive two prompts to set up passwordless ssh and remove existing MySQL installs, to which you will answer ”NO”  and “YES,” respectively.

Navigate to the install directory:

cd s9s-galera-codership-2.4.0/mysql/scripts/install/

In this directory, run the following command:

bash ./deploy.sh  2>&1  |tee cc.log

Installation completed in 790 seconds
*******************************************************************************
*                                                                             *
*                            CONGRATULATIONS                                  *
*                                                                             *
*******************************************************************************
You have now installed one of the most advanced db infrastructures available.

Open your browser to http:///cmonapi to register you cluster.
Logon with your email address and password 'admin'.
Enter this ClusterControl API token: b787f3b11462e971748f22b1313f4b68d6dd0387 when prompted.

Don't hesitate to contact us if you have any problems or inquiries. Good luck,
Severalnines AB

The partial sample output above confirms that Galera has been successfully installed on the three nodes with the default MySQL password for root as password. For now, we won’t worry about logging on to the ClusterControl.

Step 3: Transfer the data with mysqldump

Issue the following command on any of the controller nodes (l1 or l2) to backup the data:

mysqldump --routines --all-databases --events > mySQLfroml1.sql

Log on to any of the Galera nodes and dump the backed-up data with the following command:

/usr/local/mysql/bin/mysql -ppassword < mySQLfroml1.sql

Step 4: Grant Privileges

View the credentials for the MySQL connections by running the following command on any of the controller nodes:

for i in $(find /etc -name \*.conf); do grep mysql:\/ $i; done

The sample output is shown below:

sql_connection=mysql://nova:945Yo6g1jekiwBU_e3FK@192.168.236.198/nova
connection = mysql://keystone:qxyQakdR8xyOcFwj5B05@192.168.236.198/keystone
sql_connection = mysql://glance:A4uweU31SqMZM8RfiEvO@192.168.236.198/glance
sql_connection = mysql://glance:A4uweU31SqMZM8RfiEvO@192.168.236.198/glance
sql_connection = mysql://cinder:GgiLXlNnkNdILrf8CXPH@192.168.236.198/cinder

Grant privileges on the Galera Cluster to mirror the Rackspace Private Cloud MySQL install by plugging in the appropriate credentials as below. For example, for keystone it is qxyQakdR8xyOcFwj5B05. You can do this on any of the Galera nodes.

grant all on *.* to 'nova'@'192.168.236.11' identified by '945Yo6g1jekiwBU_e3FK';
grant all on *.* to 'nova'@'192.168.236.12' identified by '945Yo6g1jekiwBU_e3FK';
grant all on *.* to 'nova'@'192.168.236.21' identified by '945Yo6g1jekiwBU_e3FK';

grant all on *.* to 'keystone'@'192.168.236.11' identified by 'qxyQakdR8xyOcFwj5B05';
grant all on *.* to 'keystone'@'192.168.236.21' identified by 'qxyQakdR8xyOcFwj5B05';

grant all on *.* to 'cinder'@'192.168.236.11' identified by 'GgiLXlNnkNdILrf8CXPH';
grant all on *.* to 'cinder'@'192.168.236.21' identified by 'GgiLXlNnkNdILrf8CXPH';

grant all on *.* to 'glance'@'192.168.236.11' identified by 'A4uweU31SqMZM8RfiEvO';
grant all on *.* to 'glance'@'192.168.236.21' identified by 'A4uweU31SqMZM8RfiEvO';

grant all on *.* to 'admin'@'192.168.236.11';
grant all on *.* to 'admin'@'192.168.236.21';

grant all on *.* to 'monitoring'@'192.168.236.11';
grant all on *.* to 'monitoring'@'192.168.236.21';

grant all on *.* to 'dash'@'192.168.236.11' identified by 'dash';
grant all on *.* to 'dash'@'192.168.236.21' identified by 'dash';

Step 5: Update Configuration files

Update configuration files on both the Controller nodes.

Add the following lines to /etc/haproxy/haproxy.cfg file:

listen mysql 0.0.0.0:3306
	mode tcp
	balance roundrobin
	option tcpka
	option mysql-check user haproxy
	server mysql1 192.168.236.51:3306 weight 1
	server mysql2 192.168.236.52:3306 weight 1
	server mysql3 192.168.236.53:3306 weight 1

This allows MySQL connections to be channeled via HAProxy to the three MySQL nodes.

Edit /etc/keepalived/conf.d/script_mysql.conf file as below:

script "killall -0 mysqld"

Replace mysqld with haproxy as below:

script "killall -0 haproxy"

since the MySQL daemon won’t be running locally and we want Keepalived to monitor the health of HAproxy instead.

Step 6: Disable MySQL services on controller nodes and restart

Finally, you must disable the MySQL services installed with Rackspace Private Cloud to prevent them from starting on system restart. Edit the /etc/init/mysql.conf file commenting out the following line on both the controller nodes:

start on runlevel [2345]

When commented out, it will appear as follows:

# start on runlevel [2345]

Restart the controller nodes and Rackspace Private Cloud should be using Galera. Verify that MySQL is not running on the Controller nodes by running the following command, which should fail:

mysql

A failure should produce output similar to the following:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

Now run the nova-manage and nova commands.

nova-manage service list

Running the nova service-list command as below should yield results similar to those that were seen before installing Galera:

source openrc
nova service-list

Sample output is below:

+------------------+------+----------+---------+-------+----------------------------+
| Binary           | Host | Zone     | Status  | State | Updated_at                 |
+------------------+------+----------+---------+-------+----------------------------+
| nova-cert        | l1   | internal | enabled | up    | 2013-10-18T02:03:35.000000 |
| nova-cert        | l2   | internal | enabled | up    | 2013-10-18T02:03:33.000000 |
| nova-compute     | c1   | nova     | enabled | up    | 2013-10-18T02:03:39.000000 |
| nova-conductor   | l1   | internal | enabled | up    | 2013-10-18T02:03:41.000000 |
| nova-conductor   | l2   | internal | enabled | up    | 2013-10-18T02:03:38.000000 |
| nova-consoleauth | l1   | internal | enabled | up    | 2013-10-18T02:03:36.000000 |
| nova-consoleauth | l2   | internal | enabled | up    | 2013-10-18T02:03:33.000000 |
| nova-network     | c1   | internal | enabled | up    | 2013-10-18T02:03:38.000000 |
| nova-scheduler   | l1   | internal | enabled | up    | 2013-10-18T02:03:35.000000 |
| nova-scheduler   | l2   | internal | enabled | up    | 2013-10-18T02:03:41.000000 |
+------------------+------+----------+---------+-------+----------------------------+

OpenStack away with Galera!

Summary

HA on Rackspace Private Cloud is implemented using Keepalived, VRRP and HAProxy. It works on master-master for MySQL replication. When there are more than two MySQL nodes involved, an alternate form of HA is required.

We started with the default implementation for MySQL. With some changes to configuration files and dumping data from the existing MySQL database, we’re able to replace the standard MySQL implementation with the Galera implementation. Since Galera is based on multi-master replication there is an added benefit of achieving scale out for MySQL.

The OpenStack High Availability Guide is an excellent resource and discusses the Galera method in more detail and alternate methods of HA such as DRBD, Pacemaker and Corosync. The Rackspace Private Cloud Knowledge Center goes more in-depth into the respective installation.

Again, if you are attending the OpenStack Summit Hong Kong in November, please stop by my talk and say hi!

About the Author

This is a post written and contributed by Raghavan "Rags" Srinivas.

Raghavan "Rags" Srinivas works as a solutions architect at Rackspace where he finds himself constantly challenged from low level networking to high level application issues. His general focus area is in distributed systems, with a specialization in Cloud Computing and Big Data. He worked on Hadoop, HBase and NoSQL during its early stages. He has spoken on a variety of technical topics at conferences around the world, conducted and organized Hands-on Labs and taught graduate classes in the evening.

Rags brings with him over 20 years of hands-on software development and over 10 years of architecture and technology evangelism experience. He has evangelized and influenced the architecture of a number of technology areas. He is also a repeat JavaOne rock star speaker award winner.

Rags holds a Masters degree in Computer Science from the Center of Advanced Computer Studies at the University of Louisiana at Lafayette. He likes to hike, run and generally be outdoors, but most of all he loves to eat.


More

Leave a New Comment

(Required)


Racker Powered
©2014 Rackspace, US Inc.