EDITOR’S NOTE: Due to a global shortage of IPv4 addresses, we are no longer allowed to give out additional IPs unless they are for SSL certs. This is a Rackspace policy and we can not add an IP for any other reason.

The throughput of an individual server can be the bottleneck for almost any Cloud based server.  With restrictions placed on a shared environment, you may find that you can no longer grow your solution horizontally behind a proxy load balancer.  There are options to work around this, the first and easiest being DNS round robin load balancing. DNS round robin load balancing isn’t smart and will just keep rotating, even if a node goes down – this is a huge downside.

You’ve got other solutions though; I want to introduce you to LVS-TUN, a tunneling load balancer.  The Linux Virtual Server project has been around for a while now, and has had different iterations, but LVS-TUN seems to work best on our infrastructure.

http://www.linuxvirtualserver.org/

LVS-TUN is a tunneling load balancer solution that will take all incoming requests through the load balancer and forward the packet to the web nodes. The web nodes will then respond directly to the client without having to proxy through the Load Balancer. This type of solution can allow for geo-load balancing, but will more importantly allow a customer use the bandwidth pool available from all web nodes, instead of relying on the limited through put of the load balancer.

PREREQUISITES

• 2 x 256MB Cloud servers to serve as Load Balancers, running Centos 5.5
• 2 x 256MB Cloud servers to serve as Web Nodes, running Centos 5.5
• An additional IP on Load Balancer 1, shared with Load Balancer 2 and all Web nodes.
• All servers must be on the same Huddle.
• All servers must share an IP (explained later in this article)

INSTALLATION

Networking:

A shared IP address must be common among all Load Balancers and Web nodes in this cluster.  Also another thing to remember is that all of your servers will need to be on the same Huddle in cloud servers; the IP pools available in Cloud Servers are restricted by Huddle.

Load Balancers (aka Directors):

What to install on your Load Balancers:

• piranha

# This list is pretty short, but installing this package through YUM will install all necessary dependencies to get you working.

# Make sure to make it start on boot with “chkconfig pulse on”.

The LVS-TUN Load Balancer’s configuration will be identical on both servers.  First we need to make a couple of changes to the sysctl.conf file; these kernel parameters will allow for things like the shared IP to be bounced between Load Balancing Nodes.

sysctl.conf:

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.ip_nonlocal_bind = 1

Once you’ve made changes to this file, run ‘sysctl –p’ to make them live inside your system.

Moving on to the LVS-TUN service configuration file, which is found at /etc/sysconfig/ha/lvs.cf; this will configure both the load balancing and fail over service.   The service that will read this file is pulse, and is installed as part of piranha.  Pulse is a heartbeating daemon for monitoring the health of the nodes on your cluster.

• Remember to increment the serial number each time that you make a change to this file, or you won’t see the change go into effect.
• If you want to run HTTPS, don’t run a health check on port 443; the system doesn’t fully support health checks over HTTPS that I could find.

You’ll want to ifdown the shared IP, usually eth0:1, and then delete its configuration file in /etc/sysconfig/network-scripts/.  The pulse service will bring the IP live on the currently active load balancer, instead of the networking service used in most RPM based systems.

/etc/sysconfig/ha/lvs.cf

# serial must be incremented every time that a change
#is made to this configuration file
serial no = 1
#primary IP for your master LB, used as an identification
primary = 184.106.233.251
primary_private = 10.179.109.69
#primary IP for your slace LB, used as an indentification
backup = 184.106.233.252
backup_private = 10.179.119.61
# must be listed as active, or slave will terminate pulse
backup_active = 1
# Failover Options
heartbeat = 1
keepalive = 2
deadtime = 15
# LVS Type
service = lvs
network = tunnel
debug_level = NONE

##  Simple HTTP configuration
virtual http-pool {
active = 1
# the VIP that you want to bind to, this
# is only a forward and will not show up in a netstat
address = 172.x.x.x eth0:1
vip_nmask = 255.255.255.255
port = 80
persistent = 600
pmask = 255.255.255.0
send = "GET / HTTP/1.0\r\n\r\n"
use_regex = 0
load_monitor = none
scheduler = wrr
protocol = tcp
timeout = 6
reentry = 15
quiesce_server = 0
server web01 {
address = 10.x.x.1
active = 1
weight = 10
}
server web02 {
address = 10.x.x.2
active = 0
weight = 10
}
}

## SSL Configuration
virtual ssl-pool {
active = 1
address = 172.x.x.x eth0:1
vip_nmask = 255.255.255.255
port = 443
persistent = 600
pmask = 255.255.255.0
load_monitor = none
scheduler = wrr
protocol = tcp
timeout = 6
reentry = 15
quiesce_server = 0
server web01 {
address = 10.x.x.1
active = 1
weight = 10
}
server web02 {
address = 10.x.x.2
active = 0
weight = 10
}
}

Web Nodes configuration (aka Real Servers):

Install your Web Head as you normally would – apache, lighttpd or nginx.  It shouldn’t make a difference which http service you are running.  The main changes that we will be worried about will be to the sysctl parameters and a Tunneled IP address.  Just like the Load Balancers, let’s start by running ifdown on the VIP, eth0:1, and then deleting its configuration file.  We’ll reconfigure the IP as a tunnel on the web nodes a little later.  First let’s make a few changes to sysctl.conf; we’re doing a few things in this file.  These changes need to be made due to the default settings that come from Centos, which would normally block this type of transaction.

sysctl.conf:

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.tunl0.arp_ignore = 1
net.ipv4.conf.tunl0.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.ip_local_port_range = 1024 65536
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.eth1.rp_filter = 0
net.ipv4.conf.tunl0.rp_filter = 0

Tunnel device:

Ok, here is a very important step: this tunneled IP will allow the web-node to respond directly to client giving the Virtual IP address shared amongst the cluster.  This shared IP makes the client think that it is still talking to the load balancer, so the connection is never terminated.

/etc/sysconfig/network-scripts/tunl0:

DEVICE=tunl0
TYPE=ipip
# this is the shared IP from the Load Balancer
IPADDR=172.x.x.x
NETMASK=255.255.255.255
ONBOOT=yes

Once you’ve saved this file, you can bring up the tunnel with ‘ifup tunl0’.  It should show up in ifconfig like any other IP and it should be listed as an ‘ipip’ type.

START LOAD BALANCING:

starting:
/etc/init.d/pulse start

reloading:

/etc/init.d/pulse reload

stopping:
/etc/init.d/pulse stop

Once your load balancer is up and running, you’ll want to run ‘ipvsadm’; this will tell you which load balancer is currently active and which web-nodes are active behind that load balancer.  The active balancer will give a similar answer:

[root@lvs01 ~]# ipvsadm

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port                 Forward             Weight             ActiveConn             InActConn

TCP  174-x-x-x.static.cloud- wrr

-> 10.x.x.x:http                       Tunnel              10                             0                      0

-> 10.x.x.x:http                       Tunnel              10                             0                      0

Once everything is up and running, your load balancer should take a request and forward the packet to your web-node.  Then, the web-node will respond directly to the client.  With this type of traffic pattern, your load balancer is essentially only taking incoming traffic.  This will significantly increase its ability to handle traffic.  The outgoing traffic back to the client will be shared amongst your web-nodes.  You’ve just allowed your cluster to handle a much larger scale of traffic, since your traffic is now spread against your cluster.

Helpful Notes

• ipvsadm – this command will tell you what LVS solutions are currently running. While the server is listening on port 80, it will not show up in netstat. It’s simply forwarding that port traffic to another server. The web server itself will respond directly to the client.
• The heartbeat solution used in the configuration is pulse and not heartbeat like we would normally use. It has a slightly irregular behavior compared to heartbeat. While there is a master and slave named in the configuration file, the Master Load Balancer will not capture the IP whenever it is online. Whoever has the shared IP at the time is the true master, and there will not be a change unless there is a failure.
• piranha has a gui editor.  You can turn it off, you won’t need to use it.
• You’ll want to move sessioning to your database or a shared disk solution, if you haven’t already.
• Turn off IPtables until you’ve got everything working.

In this article, I will walk you through creating a simple software load balancer using our Cloud Servers product. This is an entry-level job using simple readily available packages from any of the distributions repositories. I’ll be using Apache as our load balancer (Apache can be used as many things including a load balancer) in conjunction with the Apache module mod_proxy and mod_proxy_balancer. Both are available through CentOS and I’ll be using this as my base install.

The main thing that I want you to take out of this is that you can use Cloud Servers to scale horizontally. You can only scale a server vertically (i.e. more memory, processor) as a web head so far and eventually you won’t receive better performance from a faster server. This is when you need horizontal expansion and will need to add drone servers behind a smart host, each working a piece of workload.

Prerequisites

Hardware

We are going to use a total of 3 boxes to start out with:

•    One 256M Cloud Server to be used as the load balancer
•    Two Cloud Servers to be used as dumb web heads

Software

The software for all three servers will be the same technically where they will be running the same packages. We’ll throw in only two software groups.

•    Update your system to all the newest goodies:

•    Apache and all its goodies, CentOS makes this ultra simple with the groupinstall feature:

•    I will be installing links to a text based web browser in case we ever need to check that a particular web head is displaying the page it is supposed to be behind the load balancer (this is optional):

Server Configuration

Web Servers

This is going to be the easiest part of the entire configuration. Since the webheads are really just drones, they’re only going to be doing grunt work and need no special configurations. That’s right, you don’t even open the httpd.conf file, just put a file called index.html in /var/www/html/index.html. In this file you can put any distinguishing characteristics you want. I put “It works, you are looking at WebHead #” where # is the numerical identifier of that particular webhead.

Load Balancer

This is the tricky part of the operation. I’ll walk through each step and then bring it together at the end for you so you know what the end product should be. All of the configurations that we are going to go through should be placed at the bottom of /etc/httpd/conf/httpd.conf in a standard Virtual Host to work.

Unwanted Requests

We are not working as a Forwarding Proxy (better known as an Open Proxy, and it’s bad news as it allows people to mask their identity by using your server to view web pages for them, it has its uses but not in this scenario). Turning off ProxyRequests will help avoid any unwanted traffic.

The Balance

In this part of the Virtual Host we will be naming our web heads and declaring how we will be balancing. The BalanceMember directive is how you declare the webheads. Of course you can add as many as you would like, using these as templates. The ProxySet directive declares how you would like to balance. We’re going to use a “byrequest” balancing algorithm which is the same as a Round Robin, so for each new request you will get a new webhead. The order is sequential and although there are better and smarter algorithms out there, this is the easiest to configure and you need no knowledge of networking theory. All of this will be wrapped in <Proxy> tags, which is how Apache knows to send it to mod_proxy. The “balancer://mycluster” identifier is only an identifier although you could technically call it what you want as long as you put the “balancer://” prefix.

Keep in mind you will want to contact your webheads from the load balancer with their private IPs (this will keep your bandwidth charges down to a minimum by keeping all inter server communication between on the private network, where bandwidth is free).

Balance Manager

Optional Step
This is a tool packaged with the mod_proxy_balancer tool and it allows you to make configurations from a gui tool through the web browser. It is viewable at “http://domain.com/balancer-manager.” Keep in mind that these changes die after you restart Apache. I won’t go over how to use this tool, but it is available to you.

ProxyPass

This is the last part of the configuration and just adds the situations that will need to be proxied. We don’t want to proxy the balancer-manager, but we do want to proxy everything else.

Summary

If you have all this in your httpd.conf on your load balancer Cloud Server and start up Apache, you should be able to view your domain name that is properly pointed to your load balancer. When you hit refresh it should hop between your two webheads saying, “It works, you are looking at WebHead 1″ or “It works, you are looking at WebHead 2.” Congratulations, you are now balancing.

What I have done for you is combine all the things we’ve learned into a helpful packaged VirtualHost. You just need to trade out all the necessary values that are specific to your configuration like the domain name and the IP addresses to your webheads. Also, there are some security additions that are explained in the comments; everything is commented so you don’t have to refer back to this article to make changes later.